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[57] ABSTRACT 

A method implemented by a computer-based electronic 
assistant to receive and manage incoming calls to a sub- 
scriber including the steps of receiving an incoming call to 
the subscriber from a caller; establishing a first connection 
between the electronic assistant and the caller, establishing 
a second connection between the electronic assistant and the 
subscriber; over the second connection, electronically noti- 
fying the subscriber of the incoming call; in response to 
receiving a call accept command from the subscriber over 
the second connection, linking the caller and the subscriber 
so that they may communicate with each other; upon linking 
the subscriber to the caller, switching the electronic assistant 
to a background mode in which said electronic assistant 
continues to monitor the subscriber over the second con- 
nection while the subscriber is linked with the caller; and in 
response to receiving a summoning command, switching the 
electronic assistant into a foreground mode, wherein the 
electronic assistant when in its background mode responds 
to a first set of commands including at least the summoning 
command and when in its foreground mode responds to a 
second set of commands where the second set of commands 
is larger than the first set of commands. 

57 Claims, 24 Drawing Sheets 
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NETWORK BASED KNOWLEDGEABLE 
ASSISTANT 

This is a continuation of application Ser. No. 08/316,635, 
filed Sep. 30, 1994 now U.S. Pat. No. 5,652,789. 

BACKGROUND OF THE INVENTION 

The present invention relates to a computer-implemented 
method and apparatus for managing communications to 
and/or from a user over multiple media. 

Today there are many different, commercially available 
devices that enable people to communicate with each other 
electronically. In addition to the ubiquitous telephone that 
has been around for decades, there now are cordless phones 
for the home, mobile phones for the car, handheld wireless 
phones which fit into a person's jacket pocket, pagers, local 
and wide area computer networks, and facsimile machines, 
to name a few. Undoubtedly, the number and type of devices 
and their sophistication will continue to increase over time. 
Indeed, it is likely that a day will soon arrive when it will be 
possible for everybody to conveniently and inexpensively be 
within arms reach of some communication device that 
enables them to communicate electronically with other 
people. 

The proliferation of different types of communication 
devices and the increasing diversity of communications 
media present new challenges. How will communications 
among the different devices and over the different commu- 
nications media be coordinated and managed so that people 
have truly effective access to each other? One challenge is 
associated with communicating information between and 
across different communications media. Another challenge 
is related to handling the inevitable increase in the number 
of calls so as to maintain accessibility of users. For example, 
as more people come to rely on their wireless phones to 
transact business while on the road or away from their 
offices, their phones are likely to be busy a larger percentage 
of the time. As a consequence, although a wireless phone can 
go anywhere with its owner, to the people trying to reach 
that owner when the phone is in use, the owner will still 
seem to be as inaccessible as when he did not carry a 
wireless phone. In addition, the more the owner of such a 
device uses it, the more likely it will be that he will not know 
that somebody else was trying to reach him and thus he may 
miss important calls. 

An obvious advantage of many of the new commercially 
available communications devices is that they offer the 
possibility of greater mobility to the user. Unfortunately, 
however, it is not always having to be near the office 
telephone that ties a business person to the office. The office 
provides other services that are also important and may not 
be so mobile. Thus, to fully realize the greater mobility that 
is offered by the new communications devices and media, 
these new technologies must be provided in a way that takes 
into account the business person's dependance on other 
services besides communications. 

SUMMARY OF THE INVENTION 

The invention described herein is referred to as an elec- 
tronic assistant. It is a computer-implemented entity that 
assists a subscriber with his or her communications by 
carrying out tasks that are delegated to it. The electronic 
assistant, modeled to have human-like qualities, recognizes 
speech and performs functions within the familiar model of 
an office. 

Each subscriber who has an account is assigned at least 
one electronic assistant which may be dedicated or shared. 
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The electronic assistant offers services to both the subscriber 
and to any contacts or other callers trying to reach the 
subscriber. A contact is a person, place, or group that the 
subscriber has described for the electronic assistant. A 

5 contact can be another subscriber or an outside caller. 
Electronic assistants treat both subscribers and contacts as 
users of the system. 

The electronic assistant offers a wide range of services to 
its subscriber, among which are the following. The elec- 

!0 tronic assistant can handle incoming calls from several of 
the subscriber's personal contacts while at the same time it 
is doing any of its tasks, including reviewing messages with 
the subscriber, managing information, etc. It can make 
logical decisions about how to manage, whether to forward, 

15 and where to forward the incoming calls from the subscrib- 
er's contacts. The electronic assistant can also communicate 
with the subscriber and the subscriber's contacts using a 
wide variety of different communication devices, such as 
telephones, fax machines, pagers, computer terminals, and 

20 communications enabled handheld devices (e.g. Personal 
Digital Assistants otherwise referred to as PDA's). When the 
subscriber tries to reach a contact or a contact tries to reach 
the subscriber, the electronic assistant mediates the connec- 
tion and then remains available to add value to the session. 

25 The electronic assistant can schedule and manage reminders 
for its subscriber. When reminders come due, the electronic 
assistant notifies the subscriber. 

In general, in one aspect, the invention is a method 
implemented by a computer-based electronic assistant to 

30 receive and manage incoming calls to a subscriber. The 
method includes the steps of: receiving an incoming call to 
the subscriber from a caller; establishing a first connection 
between the electronic assistant and the caller, establishing 
a second connection between the electronic assistant and the 

35 subscriber; over the second connection, electronically noti- 
fying the subscriber of the incoming call; in response to 
receiving a call accept command from the subscriber over 
the second connection, linking the caller and the subscriber 
so that they may communicate with each other; upon linking 

40 the subscriber to the caller, switching the electronic assistant 
to a background mode in which said electronic assistant 
continues to monitor the subscriber over the second con- 
nection while the subscriber is linked with the caller; and in 
response to receiving a summoning command, switching the 

45 electronic assistant into a foreground mode. When in the 
background mode, the electronic assistant responds to a first 
set of commands including at least the summoning com- 
mand and when in the foreground mode, it responds to a 
second set of commands. The second set of commands is 

50 larger than the first set of commands. 

In general, in another aspect, the invention is a computer- 
implemented method of processing an electronic reminder 
that is addressed to a subscriber. The electronic reminder 
includes subscriber-generated content and a specified time at 

55 which it is to be delivered to the subscriber. The method 
includes the steps of: storing the electronic reminder in an 
electronic database that is accessible to the electronic assis- 
tant; when current time coincides with the specified time, 
detecting that the stored electronic reminder has become 

60 due; in response to detecting that the stored electronic 
reminder has become due, identifying a communications 
device through which the subscriber can be reached at the 
specified time; establishing a connection to the communi- 
cations device; upon reaching an answering party through 

65 the communications device, electronically notifying the 
answering party that the call is intended for the subscriber; 
electronically informing the answering party that the 
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answering party may accept the call by issuing an accept the multi-media reference contains a plurality of references 

reply; if the call is accepted by the answering party, elec- to the item of information, each of which refers to the item 

tronically delivering the contents of the electronic reminder of information in a different one of a plurality of formats, 

tome answering party through said communications device. each of which is of a different media type; passing the 

In general, in yet another aspect, the invention is a method 5 multi-media reference to the ports attached to the channel; in 

implemented by a computer-based electronic assistant to response to receiving the multi-media reference at the 

receive and manage incoming calls to a subscriber. The attached ports, retrieving the item of information from 

method includes the steps of: receiving an incoming call to memory in a particular one of the formats identified in the 

the subscriber from a caller; in response to receiving the multi-media reference; and passing the retrieved item of 

incoming call, establishing a first connection between the 10 information to an input/output device for delivery over the 

electronic assistant and the caller; through a dialog between connected communications network. The step of retrieving 

the electronic assistant and the caller over the first ^ performed by one of the attached ports that is capable of 

connection, determining the identity of the caller; detecting processing the format of the retrieved item of information 

that the subscriber is presently interacting with the electronic an d the input/output device to which the retrieved item is 

assistant through a second separate connection; electroni- 1S passed for delivery is the input/output device that is asso- 

cally alerting the subscriber over the second connection that c i ate d with the attached port that retrieved the item of 

there is an incoming call for the subscriber; electronically information from memory. 

identifying to the subscriber the identity of the caller; In preferred embodiments, the electronic assistant enables 

monitoring the second connection for a response sent by the ^ individual to manage and entomb* his availability to 

subscriber to the electronic assistant directing the electronic 20 friendS; famil customer s and strangers 

assistant how to process the incommg call. depending upon the time of day, day of the week and his or 

In general, in still another aspect, the invention is a her nccds In addition, the electronic assistant has the 

method implemented by a computer-based electronic assis- subscriber's schedule and therefore knows where the sub- 

tant for managing information and connection resources for scr iber is, what he is doing, what his availability is, and how 

a plurality of subscribers including a first subscriber and a 2 S to reac h mm ( e . g . through what communications device), 

second subscriber. The method includes the steps of: receiv- schedule is used to manage the accessibility of the 

ing a call from the first subscriber to the electronic assistant subscriber to others and his visibility to other subscribers on 

over a communications media; establishing a first connec- me svs t em 

tion between the electronic assistant and the first subscriber; , , c ^ .„ . . c 

*j *•* • a. * u ■# - 4 . „ Other advantages and features will become apparent from 

identifying the first subscriber as the source of the call; 30 4 . * n • j • *• c r j l j- * ^ 

, . ~ , . . -j , ^ - 1! me following description of the preferred embodiment and 

starting up a first session m said electronic assistant which f rom the claims 
is a thread of execution of code for managing data and 

performing functions on behalf of the first subscriber; within BRIEF DESCRIPTION OF THE DRAWINGS 
the first session, receiving a first command sent by the first 

subscriber to the electronic assistant over the first connection 35 FIG. 1 shows the electronic assistant and office items; 

instructing the electronic assistant to perform a function FIG. 2 shows the hardware platform; 

relating to the second subscriber; responding to the first FIG 3 fe a ^^0^ overview of the software architec- 

command by sending a first message addressed to the second mre . 

subscriber and containing information relating to the first ' A . . - , - , . ^ 

j • t ( , ^ . _ FIG. 4 is a process view of the software architecture; 

command; in response to the first message, starting up a 40 Y 

second session which is a thread of execution of code for nG - 5 is a functional block diagram of the system 

managing data and performing functions on behalf of the highlighting the hardware interface to the communications 

second subscriber and which is separate from the first channels; 

session; within the second session, receiving the first mes- FIGS. 6A-H illustrate the process of completing a task 

sage and performing a function that produces a result that is 4s and the objects that are involved; 

responsive to the first message. FIG. 7 illustrates the use of the box, the hand and the 

In general, in another aspect, the invention is a computer- finger in manipulating items; 

implemented method of processing communications piG. 8 shows a sample MMUI menu; 

through a multimedia interface that includes a plurality of FIG. 9 shows a sample MMUI menu with moguls; 

interface devices and a plurality of input/output devices. 50 „« 4 „ . . 

Each of the interface devices is capable of connecting to a FIGS * ™ A ~ D show the relationship between memes, 

different one of a plurality of different communications menus and moguls; 

networks, and each of the input/output devices is capable of H illustrates the process of submitting and de liver- 
processing a different one of a plurality of media types. The ing a message and the objects that are involved; 
method includes the steps of: establishing a channel repre- 55 FIG. 12 illustrates the process of delivering reminders and 
senting a physical connection to any selected one of the the objects that are involved; 
plurality of communications networks through the interface piG. 13 shows the contents of a box; 
devices; attaching an appropriate subset of a plurality of FIG. 14 shows the contents of a user object; 
ports to the channel, wherein each port represents a different ___ « 1 »• L . *_ . 

fl , . . . . j i_ • *u •* « FIG. 15 shows the relationship between user obi ects and 

one of the input/output devices and wherein the appropriate 60 r J 

subset of ports includes ports which correspond to input/ contacts, 

output devices that are capable of connecting to the selected Fia 1* shows the flow of events between the VM and the 

communications network; executing an operation that gen- agent sessions and hardware; 

erates an item of information that is to be communicated FIG. 17 shows the VM internal objects and their relation- 
through the multimedia interface to at least one of the 65 ship to eacn other; 

communications networks; retrieving from a memory a FIGS. 18A-B illustrate the process of creating a channel 

multi-media reference to the item of information, wherein and the objects that are involved; 
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FIG. 19 illustrates the operations for presenting memes A "place" is a location with a single address (e.g. a phone 

and menus to ports; number, fax number, network id, etc.) associated with it. 

FIG. 20 shows the functional relationships between the Each person can include a standard set of places: work, 

agents and the applications services; home, car, mobile, and other. A corporation, department, or 

FIG. 21 shows the flow of data in an assistant's session; 5 ot h er organization is also considered a place, and can 

FIG. 22 shows the agent class hierarchy; include a phone number and fax number. 

FIGS. 23A-D present an example of a three-part hand A "group" is a user-defined set of persons, places, and/or 

shake; groups. 

FIGS. 24A-B is a flow chart of the answer call task; io A "message" is a piece of information that is addressed to 

FIGS. 25A-D illustrate the use of a conference object in a person or group. The most common type of message is 

establishing a call between subscribers; voice mail. 

FIG. 26 is a flow chart of the locate-and-notify task; A "reminder" is a notice to be delivered at a future time. 

FIG. 27 is a flow chart of the notify task; The reminder can be a notice to call a contact about a 

FIG. 28 shows a "Create-A-Contact" dialog with the 15 particular subject ("call reminder") or. a notice containing a 

electronic assistant; recorded audio message about another subject ("recorded 

FIG. 29 shows a "Remind-Me" dialog with the electronic reminder"), 

assistant; A "phone book" is a list of other system users that is 

FIG. 30 shows a "Remind-Me-To-CalT dialog with the 20 published for easy access by subscribers, 

electronic assistant; The "trash" is a collection of all the items that the 

FIG. 31 is a flow chart of the handle reminder task; subscriber has thrown away. 

FIG. 32 shows a "Find" command dialog; i n the described embodiment, the electronic assistant can 

FIG. 33 shows an "I- Will-Be" dialog; and recognize specific words or phrases, which are called utter- 

FIG. 34 shows a "Create -An -Itinerary" dialog. 25 ances. To recognize a subscriber's speech, the electronic 



assistant gathers training utterances from the subscriber, 
which are recordings of the way the subscriber pronounces 
a word or phrase. The process of gathering these training 



DESCRIPTION OF THE PREFERRED 
EMBODIMENTS 

The Virtual Office „ . ^ - , . ♦ • • • .u . .u 

a n * * j u nr- i , , 4 , _ utterances is sometimes referred to as training in that the 

As illustrated by FIG. 1, a subscriber works with an 30 . . .... • . . 

electronic assistant 10 using a "virtual office" as a model. 15 ™S ^ electronic assistant to recognize his 
Hie electronic assistant 10 works in an office containing the v0lce " For sample the electronic assistant may ask the 
subscriber's objects, which are called "items". An item is a subscriber to say his full name several times or the name of 
piece of information that the electronic assistant stores in a a command several times. These utterances are then corn- 
database and works on for the subscriber. The subscriber can 35 Pikd 11110 a compressed format known as a vocabulary. A 
use spoken or touch-tone commands to have the electronic "vocabulary" is a finite set of recorded words and phrases 
assistant work on various items, and the electronic assistant mat can be used directly by the system's speech recognition 
then uses a dialog to gather the information it needs from the hardware to recognize the subscriber's speech, 
subscriber to complete the task. The electronic assistant uses two different kinds of 
An item may be any one of the following: a schedule 12, 40 vocabularies, namely, a speaker-dependent vocabulary and a 
a contact 16, (e.g. a person 18, a place 22, or a group 20), speaker-independent vocabulary. The "speaker-dependent 
a message 14, a reminder 24, a phone book 26, or trash 28. vocabulary is used to recognize the speech of users who 
Messages can include other items such as a contact, have explicitly tnM tbe electronic assistant with theirown 
reminder page, etc. In derivative implementations, a mes- pronuncia ti 0 as. The speaker-independent vocabulary is used 
sage could be any sort of multimedia or composite infor- 45 to n e a of differem voices wfthout k . 

mation. u . *u 

a u u j i » i- * i_ *u • -i . . j ing each user to train the system. 

A schedule lists where the subscriber can be reached & J 

and at what time. There is a default schedule and an override In addition to these different kinds of vocabularies, the 

schedule. Tbe default schedule is the subscribers daily or electronic assistant uses two different methods to recognize 

weekly schedule; it is entered by a system administrator. 50 speech, namely, continuous recognition and discrete recog- 

Subscribers can override the default schedule with a "I-WM- nition. Continuous recognition recognizes naturally spoken 

Be" or "Create-an-Itinerary" commands to create an over- words or sequences of words, that is, words without artificial 

ride schedule (see Exhibit A at the end of the specification pauses between them. In the described embodiment, there 

for a list of spoken and touch-tone commands that an are two vocabularies recognized with continuous recogni- 

electronic assistant implements). The "I-Will-Be" command 55 ^om numbers (the digits "one" through "nine", "zero", and 

enables the subscriber to indicate when and where he can be "oh") and yes/no ("yes" and "no"). Continuous recognizers 

reached and his availability at those times. al s° discriminate against words like "urn" and "ah". Discrete 

A "contact" is the subscriber's view of a person, place, or recognition recognizes discrete, isolated words or phrases, 

group. The subscriber can use a contact as the recipient of a or sequences of distinctly spoken words that are separated 

command, such as placing a call or sending voice mail. A 60 Dv P auses - 

contact is analogous to an entry in an address book file. An The electronic assistant responds to a variety of com- 

"outside contact" is a contact that is not a subscriber. mands (see Exhibit A). Some commands operate on the 

A "person" is a representation of a person, which current context in the hand (described later), while others do 

describes another subscriber or someone outside the system. not. Some commands start a dialog between the electronic 

The description includes tbe spoken name, spelled name, 65 assistant and the subscriber so the assistant can gather the 

priority, gender, and a list of places where the person can be information it needs to carry out the task. For example, to 

reached. call a contact, the subscriber can use the "Call" command: 
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Subscriber 


Assistant 


an 


Call Whom? 


Bill Bishop 


Where? 


Work 


Dialing . . . 



Functional Description of the User Interface 

Before going into the details of the hardware and the 
software architecture of the system, a description of how the 
system handles an incoming call to a subscriber will first be 
presented. This will provide the context for then describing 
the underlying mechanisms that are implemented within the 
system to achieve their functionality. 

Throughout the remainder of this specification except 
where noted, it will be assumed that the subscriber is named 
John Smith and the caller is named Bill Bishop. 

When a caller calls into the system in an attempt to reach 
a particular subscriber (i.e., by calling a telephone number 
that is assigned to that subscriber), the system answers the 
call. By playing back audio signals that are stored in the 
system's database, the system announces to the caller: 

Good Morning, I'm the electronic assistant for John 
Smith. Please say your full name. 
The caller replies by stating his name: 

Bill Bishop 

The system records the caller's utterance and using a 
speaker dependent dictionary that is stored within the sys- 
tem's database, it attempts to recognize the caller. If the 
system cannot recognize the caller's name, it responds by 
playing the following message: 

Using touch-tones, enter your area code and phone num- 
ber followed by the # key. 
The user enters his telephone number as requested. The 
system decodes the touch tones and then searches a contact 
list for the subscriber to find a contact with that telephone 
number. 

If the system succeeds in recognizing the caller on the 
basis of his phone number, it then plays the following 
message to the caller 

Would you like to take a moment to teach me how to 
recognize your name better? 
The caller may accept this offer by pressing "9" on his 
touchstone key pad. The system then responds by playing 
the following message: 

Please repeat your full name. 
After repeating this operation a second time, the system 
stores the vocalizations with the identity of the contact. The 
next time the caller contacts the system, it will use the stored 
vocalization of the caller's name to recognize the caller. 

Once the assistant either recognizes the caller either 
through a match with a stored vocalization or through the 
caller's phone number or labels the caller as unknown, it 
then attempts to locate the subscriber. It does this by 
carrying out a sequence of operations the first of which is to 
check the subscriber's status. If the subscriber currently has 
a connection established with his assistant (and he has not 
enabled a do not disturb function), then his status is avail- 
able. If the subscriber is not connected, then the assistant 
may check a secondary information source (such as a 
cellular network) to determine the subscriber's availability. 
Finally, the assistant will check the subscriber's schedule. 
The subscriber can set his availability to indicate that he is 
accepting all calls, he is accepting no calls, or he is accepting 
only important calls. 
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If the subscriber is not accepting any calls, the system 
plays the following message to the caller: 

Sorry, he's not available. Please leave a message and then 
hang up-or press the key. Recording. 
5 After the caller has finished, the system may send a non- 
interactive notification to the subscriber that the caller has 
just tried to reach him. It does this by, for example, sending 
a page through its pager interface or sending an E-Mail 
message to the subscriber's workstation. The notification 
10 identifies the caller and it indicates whether voice mail was 
left. 

If the subscriber is accepting only important calls, the 
system checks the subscriber's contact list to determine what 
the caller's priority is. The subscriber can designate the 

15 contact as either high priority or normal priority and this 
information is stored with other information about the 
contact. If the caller has only normal priority, the system 
reports that the subscriber was not available and offers the 
caller the option to leave voice mail, as previously 

20 described. On the other hand, if the caller has high priority, 
the system continues its attempt to locate and notify the 
subscriber of the call. 

In the event that the subscriber is accepting all calls, the 
system continues its attempt to locate and notify the sub- 

25 scriber of the call without regard to the caller's priority 
designation. 

As a first step in locating the subscriber, the system 
determines whether the subscriber is already connected to 
the system, either through another call or through some other 

30 communications medium (e.g. logged into his computer). If 
the subscriber is on another call being handled by the 
system, the system briefly interrupts that call to notify the 
subscriber that he has a call waiting and it identifies the 
name of the caller. If the caller is also logged onto the system 

35 through his computer, the system may also send a visual 
message to the workstation notifying the subscriber of the 
call and identifying the caller. 

The subscriber then has the option of accepting the call, 
asking the system to place the caller on hold while he 

40 completes his present call, or asking the system to take a 
message. 

If the subscriber accepts the call, the electronic assistant 
responds by immediately establishing a connection between 
the caller and the subscriber. If the subscriber instructs the 

45 system to take a message, the system offers the caller the 
voice mail option previously mentioned. If the subscriber 
instructs the system to place the incoming call on hold, the 
system informs the caller that subscriber will be with him 
shortly. When the subscriber has completed his other call, 

50 using spoken commands, he instructs the system to then 
establish a direct connection with the new caller. 

If the system determines that the subscriber is not pres- 
ently on a call but is connected to the system or reachable 
through his workstation or other two-way data device it may 

55 send a message that is displayed on the device's display 
screen. The message includes a menu which offers the 
subscriber the option to accept or not accept the call. If the 
subscriber accepts the call, the system notifies the caller that 
the subscriber will be with him shortly. In the meantime, the 

60 subscriber calls into the system over another phone and the 
system connects him with the caller. 

If the subscriber does not accept the call or does not 
respond within some predetermined period of time, the 
system notifies the caller that it was unable to locate the 

65 caller and offers the caller the option to leave voice mail. 
If the electronic assistant does not detect the presence of 
a subscriber on the system, the electronic assistant checks 
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whether his location is indicated on one or more schedules for each AG -8 line processing card. The VPro-84 card 

that the system keeps for the subscriber. If there is a schedule performs speech recognition on commands and digits and 

that places the subscriber at a particular location at that supports up to 8 discrete recognizers or 4 continuous rec- 

moment and there are phone numbers identified with that ognizers. 

location, the electronic assistant places a call to one of the 5 The MVIP bus supports up to 256 full-duplex telephone 
phone numbers. Additionally, the assistant may check one or connections. The 256 full-duplex connections are time- 
more additional sources of subscriber location information division multiplexed (TDM) so that only 32 separate signals 
(such as cellular network databases). a re transmitted, with each signal divided into 8 time slots. 

If a party answers the call, the electronic assistant plays Interconnections between two communications channels is 

the following message to the answering party: 10 accomplished by enabling through software control each of 

Hello, I'm trying to reach Jim Smith. If he is available, the channels to have access the time slots of the other 

press the 9 key. If he is not available, press the 6 key channel. That is, the input of each channel is permitted to 

or hang up. listen to the output of the other channel. 

If the party answering the call indicates that Jim Smith is Other Possible Hardware Configurations 

available, the electronic assistant then says: is The description above presents one possible architecture 

There is a call from Bill Bishop. Do you want to take the which concentrates a number of hardware elements within a 

call? Indicate Yes by pressing the 9 key, indicate No by single computer chassis, including multiple special purpose 

pressing the 6 key. interface cards connected together with a special purpose 

If the party indicates that they will accept the call, the bus. Many other hardware organizations could also be used 

electronic assistant connects the caller to the subscriber. 20 to support the functionality described. The basic abstract 

In the event that electronic assistant is unable to establish elements needed are: (1) one or more basic computer 

a connection with the subscriber, the electronic assistant resources to support the program and data as described; (2) 

may send a non-interactive notification to the subscriber support for one or more incoming communications channels 

indicating that Bill Bishop has called him at a specified time. and support for receipt and generation of connections on the 

Hardware Architecture 25 respective channels; and (3) a switching resource to switch 

FIG. 2 shows the basic hardware components of the multiple communications channels together 

described embodiment. The system consists of a high- It is possible for these resources to be distributed across 

performance 486 computer equipped with an ISA bus 40 multiple systems. For example, in one possible implemen- 

with a passive backplane. The computer includes a CPU tation a separate switch resource could be connected to 

card 42 and display adapters (not shown). The passive 30 computers which contain hardware for managing commu- 

backplane is a standalone bus that is not part of the CPU card nications channels. These computers which manage the 

(i.e., the motherboard). A set of ISA adapters (not shown) communications channels are then connected via a network 

plug into the passive backplane to form 486 computer to larger systems which run the applications and provide 

system. Interface cards 44 and the CPU card connect directly database services. The applications and database services 

to the ISA bus. The passive backplane can hold up to 20 35 could be further split up across multiple systems, 

interface cards. In yet other hardware configurations, one might wish to 

The interface cards are special-purpose cards to support consolidate the resources even more than is described for the 

many different forms of connectivity and communication. present embodiment. For example, a single computer could 

They include network cards to connect with standard digital have a plug in card, or support on the mother board, to 

telephone lines as well as special-purpose adapters for 40 handle the communications channels. Switching could be 

recognizing speech, making phone calls, and sending and done between these channels in hardware or software. In this 

receiving faxes, etc. way the applications and database would run on a single 

A Multi-Vendor Integration Protocol (MVIP) bus 46 computer with the necessary hardware support to manage all 

consisting of a 40-pin ribbon cable is connected to all of the necessary communications channels, 

interface cards providing telephone services. The MVIP bus 45 Software Architecture 

is a high-speed communications channel that carries all This following description presents two different views of 

audio traffic between interface cards and switches telephone the system's software architecture: one emphasizing func- 

lines. tional components and the other emphasizing processes and 

Finally, the system also has fixed and removable storage events, 

including a set of high-capacity, high-speed disk drives 48 50 Functional View 

and a floppy drive 50. FIG. 3 shows the high-level software architecture of the 

Abase system, supporting 8 ports, has 2 GB of disk space system. In this and subsequent illustrations of processes, the 

and 32 MB of memory with an additional 1 GB of disk space structure objects are represented symbolically by icons 

and 8 MB of memory for each additional set of 8 ports. The which have an appearance relating to the function of the 

described embodiment, supports a maximum of 24 ports. 55 entity being represented. 

Id the described embodiment, the following specific hard- The system includes four primary components. One pri- 

ware is used. The line interface cards are either Natural mary component includes assistants 60 and agents 62. The 

MicroSystems DTI-48 Tl cards supporting connection of assistants carry out tasks on behalf of users and the agents 

two Tl trunks or Voice Technology Group Voice Bridge PC carry out tasks on behalf of assistants. Another primary 

PBX cards supporting 8 lines of PBX station set emulation. 60 component is a communication mechanism including a 

The line processing cards are Natural MicroSystems AG-8 Multimedia User Interface (MMUI) 64 and a parcel mecha- 

cards, each supporting 8 telephone channels. The ASR nism 66. These allow assistants and agents to communicate 

daughter cards are Natural MicroSystems DB-31's and there with other system components. A third primary component 

is one DB-31 card for each AG-8 line processing card. The is an object database 68 which stores user information, such 

daughter card performs speech recognition on names only. 65 as contacts and messages, and system information, such as 

Another ASR card which is provided is a VPro-84 from prompts. And the fourth primary component is a Virtual 

'Voice Processing Corporation. There is one VPro-84 card Machine (VM) 70 which services requests from assistants 
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and agents as well as hardware devices. The arrows in FIG. FIG. 5 also presents a logical view of the mechanisms 

3 show the primary paths for the flow of interactions which the system uses to connect to communications chan- 

between the various components. nels through the interface cards. To connect the system to 

The described embodiment uses the Univel UnixWare™ different communications media, the system establishes 

operating system 72 which is based on the UNIX System V 5 various channels or communications paths including a 

Release 4.2 operating system. The object database is the phone line channel 120, a fax channel 122, a TIT line 

ObjectStore™ object-oriented database by Object Design, channel 124, a two-way data line channel 126, and a batch 

}^ c ' „ „ . pager channel 128. For each channel, there is a set of ports 

Other ' Possible Software Configurations that can be attached to it. The ports, which are represented 

In the :descnbed embodiment, the VM, Assistants^gents 1Q ^ obj refer tQ deviccs 

and Database all reside on one host computer. Inere is . j *i_ ■ . * ^ 

nothing in the architecture, however, that necessitates this. supported on the interface cards. 

Other implementations could separate these components and In ' the 80 . ftware j each of the ^tunnels » represented by 

have them run on separate computers using the previously an ob ^ a data structure which identifies the physical 

mentioned remote procedure calls to communicate between Hne t0 whlch mat particular channel maps. The data struc- 

the different processes. 15 mrc includes a list of the ports which can be attached to 

Process View mc channel. Each port is represented by a port object and is 

FIG. 4 summarizes the process architecture of the system. supported by the functionality found on the interface cards 

The system consists of a Virtual Machine service process 80 that are connected to the system. That is, each port repre- 

(vmserver) and one process 82 for each instance of an sents a logical digital signal processor on the particular 

electronic assistant or agent. Each session process commu- 20 interface card which implements the functionality associated 

nicates with the vmserver using RemoteProcedureCalls with that port. 

(RPCs) 84. The vmserver services these RPCs from the FIG. 5 shows the ports that can be attached to the various 

session processes and also services events 86 from hardware channels. The phone-line channel can have a DTMF port 

devices 88. 130 ? an audio-in port 132, an audio -out port 134, and an ASR 

The Interfaces to the Communications Media 25 (automatic speech recognition) port 136. A fax channel can 

FIG. 5 is a block diagram view of the system highlighting h ave a fax-in port 138 and a fax-out port 140. Both the TTY 

the hardware interface to various communications channels. ^ channc] ^ ^ ^ ^ line channel can havc a 

A central office 90 which receives mcoming calls from text . in rt 142 and a texNmt t 144 ^ batch 

telephones 92 and fax machines 94 forwards them over a Tl , , , . . J . 

r » ti • * r j t\£. * *i_ * ™ ™. , j channel can have a text-out port 144. 

line to a Tl interface card 96 in the system. The Tl interface n , , xr , . . 

cards are connected to the MVIP bus 46 along with other 30 F ° r * e ? h ° ne hne chan » el > tbe D ™ F ^5^f the 
cards including a line processor card 98, a speech recognizer of receiving and interpreting DTMF signals sent 
card 100 and a fax board card 102. The line processing card to me svstem over the T1 bne b ? ^ caUer ^ audio-out 
does coding and decoding of speech, i.e., it plays audio that and audio-in ports represent the capability on the line 
is stored in the database and records audio for storage in the processor cards to both generate and record audio over the 
database. It also decodes touch-tone signals (DTMF). The 35 communications channel. The ASR port which is imple- 
speech recognition card performs the speech recognition mented by the speech recognition card performs the auto- 
function using a vocabulary supplied to it from the database. matic speech recognition on the audio. 
The vocabulary might be a speaker dependent vocabulary Note that throughout this description the term "call" is 
generated by the user or a speaker independent library. used in its most general sense. Not only does it include a call 
There may be one or more Tl lines coming into the 40 placed over the telephone lines but it also includes the 
system. It is likely, however, that there will be many more initiation of any contact over any of the other communica- 
phone numbers assigned to the system than there are Tl dons media including wireless communication channels, 
lines. Numbers are not mapped to particular lines. Rather, computer networks, fax channels, etc. Thus, the concept of 
when the central office receives a call intended for one of a call is not meant to be limited only to a telephone call, 
those numbers, it simply selects from among whatever lines 45 While the described embodiment includes a limited num- 
are available at the time and sends a ring signal to the system ber of channels and ports, the architecture can be expanded 
over the selected line. When the interface card for that line to handle new channels or ports providing for future forms 
answers the incoming call (i.e., connects to the line), it of connections and capabilities. 

receives from the central office a sequence of touch tones The four primary software components will now be 

(four digits) identifying the extension that is being called. 50 described in greater detail. 

Using this four-digit sequence identifying the extension and Assistants and Agents 

the subscriber's numbers stored in its database, the system The system is designed to support many different kinds of 

is able to identify the subscriber for whom the call is agents. All agents and assistants are based on a generic agent 

intended. object. Before exploring the specific agents used in the 

In the described embodiment, the system can also estab- 55 system, this generic agent will first be described, 

lish connections to a Wide Area Network (WAN) or a Local Agents, Sessions and Gadgets 

Area Network (LAN) 104 through an ethernet card 106. In An agent is a software entity that performs an action or 

addition, it can establish connections to various other com- brings about a certain result on behalf of a user or another 

munications devices through one or more serial interface agent. To communicate with users, an agent engages in 

cards 108. In FIG. 5, serial card 108 has two serial ports, one 60 dialogs using gadgets. A gadget is a representation of a 

of which is connected to a SkyTel™ system 110 over one communications device, such as a phone, fax machine or 

serial line and the other of which is connected to a two-way pager. A gadget includes a description of the communica- 

modem 112 over another serial line. The SkyTel™ system tions device as well as its address. For example, a phone 

110 transmits one way communications to pagers 114. The gadget contains the area code, number, and extension for a 

two-way modem 112 exchanges two-way communications 65 particular telephone. To communicate with a specific user 

with wireless devices 116 such as PDA's (personal digital over the telephone, an agent uses the phone gadget for that 

assistants). user. 
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To use a gadget, the agent adds it to a session. In the case 
of a phone gadget, the VM dials the phone number and 
returns an active gadget to the agent's session. An active 
gadget represents the connection from an agent's session 
through a communications network to a communications 
device. The active gadget represents a transient connection 
to the persistent gadget. An active gadget can be used for 
communicating with users, while a gadget itself cannot 
because it just stores the address and capabilities of the 
device. For example, a phone gadget specifies the number 
617-555-1212, while the active gadget represents an active 
phone line on which the number has already been dialed. A 
session consists of a collection of zero or more active 
gadgets connected together. 

To communicate with users, agents may need to add 
capabilities to active gadgets. A capability is a representation 
of an ability or feature of a device, such as the ability to 
recognize speech or play audio. For example, if an agent 
needs to fax materials to a user, it adds the fax capability to 
the active gadget. A discussion about how the VM handles 
capability requests from agents is presented later in the 
discussion of the VM. For other embodiments that include a 
graphical interface, a capability that outputs graphics and 
input capabilities that track the user's focus (i.e., mouse, eye, 
etc.) and gathers input from the user is envisioned. 

Agents can work with many different kinds of active 
gadgets simultaneously. For example, an agent can be simul- 
taneously placing an outgoing call, sending a message to a 
pager, and sending a message to a fax machine. 
User Agents, Tasks, and Presenters 

A user agent is an agent that is capable of communicating 
with humans as well as with gadgets. User agents use 
media-independent dialogs to communicate with users. 
These dialogs consist of a prompt from the user agent and a 
response from the user. When a user agent is communicating 
with a user, it directs its dialogs to an active gadget known 
as the focus gadget. For example, if a user agent is placing 
a call for the user with one active gadget and talking to the 
user with another active gadget, the latter gadget is the focus 
gadget. Other embodiments may group a set of gadgets to be 
used as the focus gadget. This would allow the assistant to 
interact with the gadget set and have the interaction span all 
gadgets in the set. 

User agents can also connect active gadgets together for 
phone conversations. When the user agent places an outgo- 
ing call for the user, and the called party answers the phone, 
the user agent connects the two active gadgets together and 
then goes in the background. When the user agent is in the 
background, it is idle; to delegate additional tasks to the user 
agent, the user needs to bring the user agent back into the 
foreground. 

A unique feature of user agents is their ability to handle 
tasks. A task is an action to be carried out by a user agent on 
behalf of a user. A task may consist of a form full of fields 
that the agent gives to a presenter. A presenter is an object 
that knows how to engage in a dialog with a user over a 
given medium as part of filling in the fields of a task. An 
agent gives each task to a presenter. The presenter gathers 
information for a field and then hands the task back to the 
agent. The agent looks to see if a field changed recently and 
examines it. For example, the agent may request information 
from the database and adjust the next field in the form. Then 
the agent hands the task back to the presenter, and it gathers 
information for the changed field. This activity of handing 
the task back and forth between the agent and the presenter 
continues until the agent is satisfied that all required fields 
are complete. Then, the agent executes the task. 
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FIGS. 6A-H illustrate the process of assigning a task and 
monitoring its progress. In this example, a user agent 150 
has obtained a focus active gadget 152 for its session 154 
and a user 156 has issued the "call" voice command (FIG. 

5 6A). In response, the agent assigns a call task 158 to a phone 
presenter 160 (FIG. 6B). The presenter interacts with the 
user over the focus gadget to fill in the fields of the task 
(FIG. 6C). The presenter gathers the name of the contact to 
call and places it in the first field before handing it back to 

10 the agent (FIG. 6D). The agent notices that this field changed 
and looks up the contact in the object database 68 (FIG. 6E). 
The agent adjusts the second field of the form to include the 
valid places defined for this contact and hands the task back 
to the presenter. The presenter gathers the location where to 

15 call the contact and places it in the second field before 
handing it back to the agent (FIGS. 6E-F). Once the fields 
of the task are filled, the user agent places the call (FIG. 6H). 

A user agent obtains the list of tasks to carry out from an 
object known as the task stack. When the agent receives 

20 parcels (to be described shortly), it may decide to place tasks 
on the task stack as a result of processing the contents of the 
parcel. The user agent gets the next task from the stack and 
carries it out. It repeats this process until there are no more 
tasks on the stack, and then it exits. In the process of carrying 

25 out a task, another task may be pushed on the stack. 

A task is not the only way that an agent interacts with a 
user. For simple interactions, the agent uses a C++ method. 
Tasks are used primarily for complex interactions. 
Assistants and Electronic Assistants 

30 An assistant is a user agent that is capable of making 
logical decisions and performing complex tasks on behalf of 
its users. The entity that will be referred to hereinafter as the 
electronic assistant is an assistant that assists users with their 
calls, messages, contacts, and schedule. Each subscriber has 

35 an electronic assistant. As an example of a complex task that 
distinguishes the Electronic Assistant from an ordinary user 
agent, the electronic assistant can locate a user by consulting 
the user's schedule and deciding which numbers to call. 
Up to this point, three different classes of agents have 

40 been discussed: agents, user agents, and assistants. Table I 
summarizes the distinctions among them. 
Distinctions Between Agents and Assistants 



45 



50 



TABLE I 


Agent Type 


Description 


Agent 


A software entity that performs an action 




or brings about a certain result on behalf 




of a user or another agent. 


User agent 


An agent capable of communicating with 




humans using media-independent dialogs and 




carrying out tasks on a user's behalf. 


Assistant 


A user agent capable of making logical 




decisions and performing complex tasks. 



55 Any agent can logically run several sessions simulta- 
neously. Each time an agent handles a session, there is a 
separate process running for each session. The master ses- 
sion is the session in which the electronic assistant is talking 
to its subscriber or trying to locate its subscriber. There can 

60 never be more than one master session running per sub- 
scriber. 

Box, Hand and Finger 

Referring to FIG. 7, the electronic assistant uses three 
different objects to manipulate its user's items, namely, a 
65 box 170, a hand 180, and a finger 182. The box is an object 
that contains all the items belonging to a user. The hand is 
an object that holds the items that the user is manipulating. 
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The finger is a software pointer that marks the currently For example, let's assume the assistant is holding and 

selected item in the hand. pointing to a contact named Bill Bishop and the subscriber 

When the subscriber asks the electronic assistant to thc following commands: 
"Find" items, such as contacts, the electronic assistant looks 

in the box for the items and then picks them up in its hand. 5 ; 

The hand can pick up a subset of the items in a box Find <findwhat> 

according to certain criteria. For example, when the sub- New-Message <one new message from Bill Bishop> 

scriber asks the electro nic assistant to find saved messages, What's-it-Say 

it only finds the messages that are marked with a saved flag. GoBack 

When the user asks for the first item, the electronic 10 

assistant moves its finger to that item and selects it. If the In this example, the subscriber replaced the contents of the 

user asks for the next item, the electronic assistant moves its Dand (which had been holding a contact for Bill Bishop) 

finger to the next item in its hand. The items in the hand can with a new message from Bill Bishop. After listening to the 

be accessed in a circular fashion. If the finger points to the message (as a result of issuing the What's-It-Say command) 

last item in the hand and the user asks for the next item, the 15 subscriber said Go-Back The result of this command is 

finger moves around to the first item in the hand. that the previous state of the hand (the single contact for Bill 

_ „ „ w „ . Bishop) is restored. The subscriber can now manipulate this 

Sequential vs. Random Access to Lists of Information contact for Bill Bishop 

Using first-item and next-item provides the ability to Another useful tool for managing state is the ability to 

browse a set of items in a sequential fashion. So, if a 2 o refer to the current item being pointed to in dialogs. The 
subscriber instructed their assistant to Find New-Message, utterance "This-One" is used to refer to an object currently 
then using next item and previous item would give the being pointed to by the assistant in the hand. The item is also 
ability to traverse through the new messages in forward or referred to as "it" so that commands such as "Describe-If, 
reverse order. First item would return the finger to point to "Update-It", "Throw-It-Away" also refer to the current item, 
the beginning of the list. 25 Finally, the commands "Send-A-Copy", "Send-A-Reply" 

In addition to this form of sequential navigation through and "Give-Them-A-Call" are operations on the current items 
items in a list, the system also provides capabilities to access to which the assistant's finger is pointing. These draw on the 
information randomly or based on more complex queries. context and leverage on a shared understanding between the 
For instance, a subscriber can ask to see all the new message subscriber and the assistant of the context. The result is a set 
from a particular contact. The following dialog: 30 of condensed dialogs that do not need to explicitly refer to 

objects which can be inferred. Other embodiments can take 
advantage of pronouns such as "him", "her" and "them" to 

refer to objects that have recently been referenced. 

Find <find wnat> Parcels 

New-Messages-From <new message from whom> . 

Bill-Bishop 35 Agents can communicate with each other using parcels. A 

parcel contains the address of the sender (From), the address 

of the recipient (To), and contents (a persistent object). The 

would put into the hand all messages that had been received contents of a parcel can also be another parcel. 

from the contact named Bill Bishop. This feature can also be The VM guarantees delivery of parcels. Any parcels 

extended to the group items which a subscriber can create. 40 addressed to a user are delivered to the master session for 

As a result, if a subscriber has a group called Hot-Prospects, mat user. If no master session exists, the VM starts one. 

the above dialog could be repeated as: Then, the electronic assistant tries to locate the user from 

within the master session. In other embodiments, parcel 

addresses can support distributed boxes and may support 

„. . . . 45 communication between remote agents. 

Find <nnd what> K _ ° 

New-Messages-From <new message from whom> Agent to Agent Communication 

Hot- Prospects The fact that the VM acts as the routing and delivery 

mechanism for parcels is powerful. It allows all agent to 

„ . . , , agent communications to be mediated through a mechanism 

and the result would be a list of items that includes new 5Q ^ ntees me that a reci ient of a d ^ 

messages from members of tbe group Hot-Prospects In be either located or slafted tQ receive ^ { }q ^ 

addition to searching for items from a particular contact, this described embodimentj the ent to t ^^^00 

technique can be used to fetch items that have been stamped ^ as OTmmunication ^wecn two agents on a single 

as priority or urgent, etc. Finally, all of the items described box In Qther embodiments which support distributed 

herein can be randomly searched for and collected into the 55 systemSj ^ ^ caQ uge routing mformation embed ded in 

hand based on different attributes. the ^ addfess tQ route the parcel tQ the appropriate remote 

Context or clustered system. An agent running on one coast could 

The ability to Find items and gather them into the hand check to see if an agent for a user across the country wants 

introduces into the system a concept of state. This state to accept a call, and all the messaging could be happening 

includes the current contents of the hand and what is being 60 across SS7 (Signaling System-7 — a phone network) or some 

pointed to. Since some number of commands may need to be other wide area network without the need to do a call setup, 

issued to change the state, and since it is often desirable to MMUI 

go back to one of the previous state, the system supports an The MMUI is a media-independent interface for commu- 
ability to "Go-Back" to the previous states. Each time a nicating with users. The MMUI allows agents to focus on the 
change in the state of the assistant occurs, the new state is 65 content to be communicated rather than the format of the 
stacked on top of the previous state. Issuing the Go-Back content and the details of using specific devices in present- 
command pops the previous state of the stack. ing content. 
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The fundamental building block of the MMUI is an object 
referred to herein as a meme. A meme is a media- 
independent reference to a piece of information. The meme 
contains a set of media objects which store the information 
in a variety of different formats. A media object is a piece of 
information, such as a sound, text string, or DTMF 
sequence, that can be presented to a particular kind of 
communications device. 

Agents use memes and menus as part of their dialogs with 
users. These dialogs can be part of a task or outside of a task, 
such as from a C++ method. For example, as part of a call 
task, an electronic assistant needs to ask the user who to call. 
Within a field of the call task is a "Call Whom?" meme, 
which the presenter gives to the active gadget. The "Call 
Whom?" meme consists of several different media objects: 
for example, an audio recording for use on the telephone, or 
a text string for displaying on a computer monitor. The 
active gadget passes the meme to the VM, which selects the 
type of media based on the capabilities of the active gadget. 
For example, if the active gadget is a telephone, the VM 
selects the audio recording in the meme, and the user hears 
something like "Call Whom?". 

The type of gadget is the primary factor but not the only 
factor that affects what type of media is used from a given 
meme. Users can set preferences for their electronic 
assistants, and system administrators and system integrators 
can set system-wide preferences that apply to all electronic 
assistants. For example, a user may select verbose prompts 
and a male voice for an electronic assistant The system 
integrator may have set up the system with Spanish as the 
default language. These preferences are known as attributes 
in the MMUL An attribute is a name/value pair. Typically, an 
attribute is used to tailor the behavior of an active gadget and 
of all MMUI elements. 

When an electronic assistant obtains an active gadget 
from the VM, it sets the attributes on it. Later on in the 
session, when the electronic assistant sends memes to the 
gadget (in this example, as part of a task), the attributes and 
the type of gadget determine what media object in a meme 
is used. For example, in a bilingual voice system, memes 
may have two audio media objects: one with the information 
recorded in French, and another with it recorded in English. 
If the user had set the language attribute to French and the 
gender attribute to male in the previous example, the result- 
ing prompt would be spoken in French with a male voice. 

Another important building block in the MMUI is the 
midget. A midget is an object used for constructing a 
multimedia dialog. A midget in a multimedia user interface 
is analogous to a widget in a graphical user interface. The 
most common type of midget is a menu. Referring to FIG. 
8, a menu 180 is a set of choices that can be presented to a 
user using multiple media. A menu consists of rows describ- 
ing each choice (a meme 182) and columns describing the 
media 184 that can be used to present the meme in each row. 
An additional column optionally stores a pointer 186 to the 
referenced data object in the database e.g. a contact object. 
In some cases, columns are blank if it is not possible or 
appropriate to present the meme using that type of media. 

In a typical dialog, the electronic assistant passes a meme 
and a menu to the active gadget. The meme is presented to 
the user, the user's response is matched against the rows in 
the menu, and the menu row selected (known as the menu 
pick) and the data pointer (if present) are returned to the 
electronic assistant. For example, in the case of the call task, 
the electronic assistant passes a meme for presenting the 
question "Call Whom?" and a menu containing the user's 
contact list. In this example, the MMUI menu has four 
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columns. The first column contains the text spelling of the 
contact name; the second column contains a string describ- 
ing the DTMF sequence that can be used as a shortcut for the 
contact name; the third column contains the vocabulary used 
5 for recognizing the spoken contact name; and the fourth 
column contains a pointer to the contact's information in the 
database. 

Although this example menu has four columns, menus 
can have more columns to accommodate different attributes. 

10 For example, in a command menu, there could be verbose 
and terse versions of the spoken command or versions 
spoken in different languages. 

Some menus contain additional objects, called moguls, 
attached to them for managing complex media types. A 

15 mogul is an object that manages a particular type of media. 
A mogul can store media-specific information. For example, 
an audio mogul is attached to the audio media column of a 
menu and stores the speech-recognition vocabulary to be 
downloaded on the recognizer hardware. A mogul can 

20 update the content of certain media in the memes in a menu 
when updates occur to other media in the memes. for 
example, a DTMF mogul updates DTMF media objects 
whenever their text media counterparts change. In FIG. 8, if 
the subscriber changes the spelling of Susan Schmidt to 

25 Susan Smith, the DTMF mogul would change "773" in 
column two of the menu to "776". Media that are updated by 
a mogul instead of created by the user are called mogul- 
generated media. DTMF media is an example of mogul- 
generated media. 

30 The algorithm to generate DTMF for a name or command 
is as follows: DTMF commands are usually at least three 
digits. If there is one word, then map the first three letters to 
the corresponding telephone key that has those letters. If 
there are two words then map the first letter of the first word 

35 and use the two first letters of the second word. If there are 
three words then take the first letter of each word. In the case 
where three letters is not enough because the command is 
still not unique, then continue to take the first letter of 
subsequent words until you have a unique sequence. 

40 FIG. 9 shows the contact menu with the DTMF mogul 
188 and audio mogul 190 attached to it. Moguls form a sort 
of third dimension to menus; they can be thought of as the 
depth of a menu. 

FIGS. 10A-D show the "Call Whom?" example previ- 

45 ously discussed. First, the presenter 160 passes a meme 182 
and a menu 194 to the active gadget 152 (FIG. 10A). The 
menu has two moguls attached to it: a DTMF mogul 196 and 
an audio mogul 198. When the VM receives the menu, it 
unpacks the vocabulary stored in the audio mogul and 

50 downloads it on the recognizer hardware. Since the active 
gadget is a telephone with speech recognition capabilities, 
the subscriber hears the electronic assistant say "Call 
Whom?" (FIG. 10B). When the subscriber responds with 
"Susan Schmidt" (FIG. 10C), the recognizer hardware uses 

55 the vocabulary to find a match in the contact menu and 
returns the menu pick to the electronic assistant (FIG. 10D). 

Memes are often strung together in a meme list to form a 
complete statement. The agent or presenter can pass a single 
meme or a meme list to the active gadget. When a meme list 

60 is passed to the active gadgets the memes are played in the 
order in which they are listed. 
Utility Agents 

The parcel mechanism provides a way of communicating 
between agents and sessions. Utility agents carry out actions 
65 on behalf of the electronic assistant, such as delivering 
messages, scheduling reminders, and answering unassigned 
phone lines. There are several different types of utility 
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agents including a postmaster agent, various courier agents, 
a cron agent and a secret agent. 

The postmaster agent is an agent that receives messages 
from electronic assistants and distributes them to appropriate 
courier agents for delivery. The courier agents are agents 5 
that receive messages from the postmaster agent and deliver 
them to their destination. The message store is a portion of 
the object database containing the incoming and outgoing 
messages for all persons. 

Referring to FIG. 11, when Susan's electronic assistant 10 
156 submits a message 200 for delivery, the parcel mecha- 
nism packages a reference to the message in a parcel 202 and 
sends it to a postmaster agent 204. The parcel mechanism 
also inserts the message into the appropriate user's section 
of the message store 206. In return, the postmaster agent 204 15 
packages a chit 208 in a parcel 210 and sends it to the 
electronic assistant. A chit is a receipt issued by the post- 
master agent, which an electronic assistant can use to check 
on the delivery status of the message. 

As the postmaster agent receives messages for delivery, it 20 
selects the appropriate courier agent 212 that should deliver 
it. Different gadgets require different types of courier agents. 
For example, the system courier agent delivers messages for 
electronic assistants, and a SkyTel™ courier agent delivers 
messages to SkyTel™ pagers. When the postmaster agent 25 
gives a message to the system courier agent, it sends a "wake 
up" parcel 214 to John's electronic assistant 216 to notify it 
about the new message. The user's electronic assistant 
retrieves the message from the message store. 

If the courier agent could not deliver the message, it 30 
returns the parcel containing the reference to the message 
back to the postmaster. Depending on the number of retries 
permitted for the message, the postmaster may ask the 
courier agent to retry the delivery or mark the delivery as 
failed. 35 
Cron Agent 

The cron agent is an agent that receives reminders from 
agents, tracks them until they are scheduled for delivery, and 
then delivers them to agents. The cron agent may also track 
other items. Therefore, the parcel can actually contain any 40 
database object, not just a reminder. 

Referring to FIG. 12, when an electronic assistant 150 
submits a reminder for delivery, it creates a parcel 220 
containing a reminder 222, and the parcel is both from the 
user and to the user. Next, it takes this parcel and places it 45 
in another parcel 224, which is addressed to the cron agent 
from the user. Ihis latter parcel is submitted to a cron agent 
226. When the cron agent receives this parcel, it unpacks the 
parcel inside it. It then places the parcel in a queue 228 in 
time-sorted order with other parcels and keeps track of when 50 
this reminder should be delivered. When the time comes, 
sends this inner parcel back to the electronic assistant. 
Object Database 

The object database is an object-oriented database, i.e., it 
is a database that maintains object structures and relation- 55 
ships directly rather than flattening them and reconstructing 
them. The object database stores all the information that 
both users and the system need across sessions. For 
example, if a user creates a new contact, the electronic 
assistant stores it in the object database. When the electronic 60 
assistant speaks to a user, it is using prerecorded media 
objects that are stored in the object database. 

The object database stores a user's information using 
three different objects: a box object, a message store object 
and a user object. The box object and the message store were 65 
previously mentioned. The user object describes a sub- 
scriber or person. 
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Referring to FIG. 13, for each subscriber's assistant, the 
object database stores a box 230 containing: a pointer 232 to 
the user object 234 describing that subscriber; a contact list 
236 which is a MMUI menu containing all of the user's 
contacts; and a pointer 238 to the section of the message 
store 206 containing the user's messages. The contact list is 
used with electronic assistant commands, such as "Call" and 
"Send Voice Mail", which expect the user to specify a 
contact. 

A stored contact can also include a pointer to a note (e.g. 
a voice message) which the subscriber can generate and 
attach to the contact. When the subscriber instructs his 
electronic assistant to call the contact, the subscriber can 
also have the electronic assistant play back the attached note 
while the electronic assistant is attempting to establish the 
connection. The note might include information about the 
contact which the subscriber wishes to be reminded of 
whenever he calls that contact. For example, he may wish to 
know the name of contact's secretary so that he can address 
her by name if she answers the phone. 

Referring to FIG. 14, the database maintains a single 
definition for each user in the system, which is known as a 
user object 234. The user object specifies the user's 
password, gender, and schedule. It also contains a pointer 
240 to the contact list 236 in the box 230, a list of places 242 
(i.e., home, work, car, mobile, other, pager), a list of groups 
244 and a list of any reminders 246 that the subscriber has 
generated. 

In the database, a person contact consists of a reference to 
a user object plus local information, such as the way the user 
pronounces the contact's name and a priority. Therefore, not 
only does the box point to a user object, but each person 
contact in the contact list in a box points to a user object. A 
group contact consists of a set of object IDs for user objects. 
A place contact consists of a phone gadget and fax gadget. 

Referring to FIG. 15, the object database also contains 
phone book objects 250. A phone book is a list of other 
subscribers on the system and consists of a set of pointers 
252 to user objects. There may be one or more phone books, 
but any subscriber can access only one phone book. Each 
entry in a phone book is a reference to one of the user 
objects. Likewise, a user's contact 254 consists of a refer- 
ence 256 to a user object plus some local information. 

When a user changes the portion of a contact that is stored 
in the user object, such as the work or home phone number, 
all phone books and contacts referencing that user object 
show the new information. 

For every user item, the object database also keeps 
general information necessary for accessing that item. This 
information includes the object ID, object label, owner, time 
of last modification, time of last access, and time of last 
change to ownership. The object label is a meme that can 
include the spoken and spelled names of the item. Each item 
also has a set of associated flags, such as read, unread, 
important, and not important. 
Virtual Machine 

The Virtual Machine (VM) is the system's operating 
system, it is a process that allocates and manages system 
resources for agents and assistants. The VM responds to 
requests from agent sessions (just as an operating system 
responds to system calls) and events generated by the 
hardware. In essence, the VM is a large event processor. The 
VM initiates all I/O operations, including input (recording 
memes), a output (playing memes), and recognition 
(recognizing speech or DTMF). In addition, the VM initiates 
connections to gadgets, disconnects connections to gadgets, 
and handles incoming connections. 
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Handling Events 

Referring to FIG. 16, the VM 70 responds to three kinds 
of events: interprocess communication (IPC) messages 260 
from agent sessions 262; hardware events 264 from the 
communications cards and other hardware 266; and timer 
events 268 from its internal time queue. 

Each action taken by an agent results in an IPC message 
to the VM. The VM receives this "event" and carries out the 
request. The VM may send IPC messages back to the agent 
session to communicate with it. The VM transmits parcels 
using IPC messages. For example, when an agent sends a 
parcel to another agent, the first agent sends an IPC message 
to the VM, and as a result the VM forwards the IPC message 
on to the second agent. 

Output to and input from the hardware causes a hardware 
event. For example, when a caller begins speaking a hard- 
ware event occurs. 

Objects internal to the VM can set timeouts as part of 
handling other events. These events are placed on a time 
queue, and when the time expires, a timer event occurs that 
the VM handles like any other event. For example, when a 
user is pressing DTMF tones, a timer goes off if there is a 
long pause between tones. 

The VM must handle events quickly and efficiently to 
provide fast response time to agent requests. The VM listens 
for events and when one occurs, it blocks all other activity 
and responds to it. When it finishes handling an event, it 
listens for the next event. 

After the VM creates a session, the session sends an IPC 
connection request to the VM. A VMListener object 310 in 
the VM receives this connection request, and the VM creates 
a VMServer object 312 for that session to use in commu- 
nicating with the VM. It is the VMServer object that receives 
the IPC messages from the agent's session, as shown in FIG. 
17. The VMServer object takes the object ID and performs 
an algorithm on it to obtain the appropriate RPC target 
(VMSession 314 or VMChannel 280) in the VM. The 
VMServer object then passes on the message to the right 
VMSession or VMChannel. To do so, the VMServer calls 
the RPC target's do_rpc virtual member function, and the 
RPC target processes the message and carries out the opera- 
tion. 

Each agent session has a corresponding VMSession 
object 314 in the VM. The VMSession object stores the 
current state of the session. This object also queues parcels 
destined for the agent when there is no session running for 
the agent. Once a session is running again, it sends the 
parcels. 

Any VM object can create a Timer object to set a timeout. 
The VM stores Timer objects on its timer queue 288, and 
when the timer goes off, a timer event occurs, which the VM 
handles like any other event. 
Managing Resources 

The VM manages several kinds of resources: those that 
can be directly manipulated by agents, those that can be 
indirectly manipulated by agents, and those internal to the 
VM. FIG. 17 shows these resources in the VM where there 
are two active agent sessions and a conference between the 
sessions. 

Agents can directly manipulate sessions, channels, 
conferences, and parcels. For each agent session, the VM 
maintains a communications channel for IPC messages and 
an object that keeps track of the state of the session. 

Each active gadget in an agent session corresponds to a 
channel 280 in the VM. As previously noted, a channel can 
have a set of ports 282 attached to it, with each port 
providing one or more capabilities. A capability is a repre- 
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sentation of an ability or feature or a channel, such as the 
ability to recognize speech or play audio. Agents request 
capabilities from the VM in order to communicate with their 
users. In response, the channel attaches one or more ports, 
5 where each capability corresponds to one or more ports. 

A conference 284 is an object that can connect multiple 
channels in a single session or in multiple sessions. For 
example, to arrange a conference call, an electronic assistant 
10 places calls to each participant and then uses a conference 
object to connect the participants* channels into a single 
session. 

For each parcel sent by an agent, the VM transmits it and 
15 if necessary, queues it 

Through the manipulation of capabilities agents can indi- 
rectly manipulate ports. A port is an object referring to an 
input/output device. A channel creates one or more ports for 

20 each capability requested by the electronic assistant and 
manages the ports throughout the connection. The port 
connects to the hardware device 286 that actually has the 
capability. Ports can also be thought of as filters. For 
example, a speech recognition port is not an audio port; it 

25 takes audio and then converts it (i.e., filters it) to a menu 
selection. 

The VM also has internal resources including timers 288, 
a bus 300 and user info objects 302. A timer is an object that 
represents a timeout. For example, a timer object is set for 
the time between DTMF keys and the time between retriev- 
ing a parcel and actually receiving it. A bus is an object that 
represents a data and control hardware bus, such as an MVIP 
3S bus or a TCP/IP bus. All objects connected to the bus can 
communicate with each other using the same protocol. A 
user info object is a cached pointer to the User object in the 
object database. A user info object is created for each user 
that calls their assistants and logs in. 

40 

The VM allocates and deallocates sessions as they are 
needed. When an incoming call arrives or a parcel is 
delivered to a user that does not already have a master 
session, the VM starts a new session. 

45 

A resource manager in the VM performs resource man- 
agement functions. For example, it keeps a pool of free and 
available recognizers and interactive-in and interactive-out 
resources 304. It manages the assignment and deallocation 

so of these resources and it notifies requesters when resources 
(e.g. channels) are not available. It also holds onto reserva- 
tions for resources when they are not currently available and 
as soon as a requested resource becomes available, it assigns 
that resource to the requesting session. It also deallocates the 

55 interactive-in and interactive -out ports and recognition ports 
when they are no longer needed for a channel, freeing them 
up for use by other channels. 

Conference Objects 

60 

Conference objects are used by an agent and the VM to 
connect together multiple gadgets and their respective chan- 
nels. The agent and the VM have slightly different views of 
a conference object. The agents see only the active gadgets 
65 that the agent is in control of; while the VM sees all the 
active gadgets and their respective channels that are con- 
nected into a conference. 
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The methods on conference 
described in the following table. 



24 



objects are listed and 



Method 



Description 



SwitcbOut 



Switchln 



CreateConference 
ChangeConfeience 



DestroyConference 



This is used to add an active gadget or a 
gadget into the conference. When a gadget 
is added, an active gadget is returned. 
This removes an active gadget from a 
conference. 

This switches an active gadget out of the 
communications paths for the conference. 
The gadget remains part of the conference 
but it is not connected to the streams of 
communication with other participants. 
This allows an ageat to carry on a dialog 
with a specific active gadget without other 
participants of the conference hearing. 
This is the complement of SwitchOuL It 
switches a switched-out active gadget back 
into the media streams of the conference 
from which the active gadget had previously 
been removed. 

This creates a new conference. 

All conference operations work on a current 

active conference. ChangeConference 

changes the active conference. 

This destroys a conference and deallocates 

all resources associated with that 

conference. 



In the described embodiment, active gadgets are always 
connected to a conference object. In the case that there is 
only one active gadget, such as when a subscriber or in caller 
are conversing with an electronic assistant, then the active 
gadget is connected to a special form of conference called 
the idle conference. 
Allocating Channels and Ports 

To speed up the allocation of sessions, the VM keeps a 
session pool 308 containing sessions that are not yet 
assigned to users. As noted earlier, the electronic assistant 
handles an incoming caller using a new session. The VM 
allocated this session from its session pool. The VM creates 
the session pool at system startup time and refills it as 
necessary. Each session in the pool is for a specific type of 
agent. 

The VM also maintains channel resources, which are 
pools of channels or ports that the VM can allocate to active 
gadgets. The channel resource or port resource is responsible 
for managing the allocations and deallocations from its pool. 

In the described embodiment, upon start up of the system, 
the VM constructs the channels that are necessary to handle 
communications through the interfaces supported by the 
system. The channels which are constructed have three 
possible modes: a listen mode, an idle mode, or an outgoing 
call mode. In the listen mode, the associated interface card 
is monitoring the incoming line (e.g. the TL line) for 
incoming calls. In the outgoing mode, the interface card is 
set up to initiate a connection to its communications channel. 
Idle mode is used during transitions. Upon start up, all of the 
channels that are constructed are put into a listen mode, 
ready to receive an incoming call. 

As illustrated in FIGS. 18A-B, when an agent 150 adds 
a gadget 336 to a session, the VM gets a channel 280 (FIG. 
18A), makes a connection on the channel, and creates an 
active gadget 152 (FIG. 18B) that refers to the channel. The 
channel represents the actual connection used to communi- 
cate with the user. 

Agents can add input, output, and recognition capabilities 
to gadgets, and the VM attaches ports with those capabilities 
to the channel. There are three types of ports: input ports, 
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output ports, and recognition ports. An input port records 
audio into a meme. An output port plays audSo from a meme. 
A recognition port uses input from the user to make a 
selection from a menu. 

Each port represents a capability of a physical device, 
such as a line processing card or a voice recognition inter- 
face card. When the VM attaches ports for a given capability, 
it usually attaches more than one port. For example, in the 
described embodiment when an agent requests the recogni- 
tion capability, the VM adds a voice recognition port for 
recognizing speech and a DTMF recognition port for rec- 
ognizing DTMF tones. When the agent requests the output 
capability, the VM adds multiple output ports, with each port 
handling a different audio format. This allows transparent 
support for hardware which uses different audio formats. 
Presenting Memes and Menus 

When the agent prompts the user with memes and menus, 
each port responds to the media or mogul that it knows how 
to play or recognize. For example, the output port plays any 
audio media for system prompts, while the recognize port 
performs speech recognition to determine the user's selec- 
tion from a menu. For each channel, only one type of media 
is presented from each meme. When presenting memes, only 
one output port can present a meme at a time. When 
analyzing a response, however, multiple recognition ports 
can be listening at the same time; for example, one can be 
listening for speech and the other for DTMF tones. If DTMF 
tones occur, the DTMF recognition port cancels the speech 
recognition port that was listening for speech. Likewise, if 
speech occurs first, it cancels the DTMF recognition port. 

For example, consider the "Call Whom?" meme previ- 
ously discussed. Referring to FIG. 19, when the presenter 
160 passes the "Call Whom" meme 192 and contact menu 
194 to the active gadget 152, the VM presents the meme to 
the collection of ports 282 attached to the channel 280. The 
audio output port knows how to play this meme, so it takes 
the meme and plays it. Next, the VM presents the menu to 
the collection of ports attached to the channel. Both the 
DTMF and VPC recognition ports know how to use this 
menu, so they take it and begin listening for DTMF tones or 
speech (respectively). Once the VPC recognition port 
detects speech, it cancels the DTMF recognition port, 
matches the speech against the menu, and returns the menu 
pick to the presenter. 

When the VM presents a meme list to a collection of 
ports, each output port looks at the first meme in the list and 
any successive memes that have the same audio format. If an 
output port can play the memes, it removes them from the 
meme list, plays the memes in sequence, and returns the 
remainder of the list to the VM. The VM presents the 
remainder of the fist to the ports again, and the process 
continues as before until there are no memes left on the 
meme list. 
Switching Channels 

The VM is also responsible for connecting and switching 
calls. It keeps track of all connections between gadgets and 
it uses the conference object to connect and switch calls. A 
conference object can connect multiple channels into a 
single session. For example, to arrange a conference call, an 
electronic assistant places calls to each participant and then 
uses the conference object to connect the participants' 
channels into a single session. 
Detailed Architecture and Internal Design of Agents 

The system agents use a variety of application services to 
carry out its duties. FIG. 20 shows the functional relation- 
ships between the agents and these application services. 

Tasks 350 are part of an assistant layer 352 but are not 
used by the utility agents 354. The assistant uses the MMUI 
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356, parcels 358, database utilities 360, and miscellaneous 
utilities 362, which in turn depend on the VM 364 and object 
database 366. Database utilities are a set of macros for 
accessing the database, while miscellaneous utilities are a set 
of standard data structures and containers (such as strings 
and bit arrays). The utility agents use parcels and database 
utilities and other VM services (e.g. access to channels, 
gadgets and active gadgets) carry out their duties. The parcel 
mechanism is layered on top of both the VM and the 
database utilities. Although the functional diagram shows 
the agents relationship with the rest of the architecture, it 
does not show the dynamics of a live session. 

FIG. 21 shows a sample assistant session. At the bottom 
of the session, the MMUI, VM and parcel libraries 368, 370 
and 372 are linked in. The active gadget 374, which the 
assistant uses to communicate with the user, is layered 
directly on top of the channel 376 itself. When the assistant 
takes a task 377 off the task list 378, the task and presenter 
380 may pass menus 382 and memes 384 to the active 
channel or send parcels. In either case, the VM library sends 
an IPC message 386 to the VM, and the VM carries out the 
action. As part of carrying out the assistant's actions, the VM 
may send IPC messages back to the session. The VM library 
converts these messages back to the objects familiar to the 
assistant. The task may return menu picks, memes, and 
parcels to the assistant. 

As noted earlier, conceptually, there are three kinds of 
agents, namely, agents, user agents, and assistants. In the 
class hierarchy, however, only agents and assistants have 
classes; user agents do not have a class. 

The Agent class 398 is the base class for all agents and 
assistants. FIG. 22 shows the Agent class and other closely 
related classes, including, Session, ActiveGadget, Parcel, 
and Task classes. Also, there is a PersistAgent class, which 
holds the persistent data for an agent. (In FIG. 22, solid lines 
represent subclasses and dashed Lines represent friend 
classes.) 

Generally, the most important operations that are carried 
out by an agent are performed on the Session, Agent, or 
ActiveGadget, as outlined in Table 3. 

TABLE 3 

Class Operations Performed 

Session Manipulating gadgets 

Handling parcels 
Agent Handling tasks 

Handling contents of parcels 
ActiveGadget Presentation of prompts 

Collection of responses 

Manipulating capabilities 

Training 



The following subsections describe SessionAgent, 
ActiveGadget, PersistAgent, Task, Parcel, and ParcelCon- 
tents classes in more detail. 
The Session Gass 

The Session class is a subclass of the Client class, which 
provides it with the capability of being an RPC client in the 
VM. The session provides the thread of execution for the 
agent and keeps track of the Interprocess Communication 
(IPC) connection to the vmserver process. 

The Session class consists of the following data members: 

A socket, which is used for communicating with the 
vmserver process (inherited from Client class) 

An object ID (inherited from Client class) 

A pointer to itself (inherited from Client class) 

A reference to the user object 
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A pointer to the focus gadget 
A list of gadgets 
A list of parcels 
The Session class contains member functions for: 

Managing gadgets. The set includes functions for adding, 
removing, and connecting gadgets and getting and 
setting the focus gadget. 
Handling parcels. The set includes functions for sending 
!0 parcels, retrieving parcels, replying to parcels, and 
queueing parcels. 
Making a master session out of the current session. 
The Agent Class 

The Agent class is a subclass of both the Typed and 
15 ParcelHandler classes. The Typed class is the primary base 
class. It provides run-time checking and member functions 
for downcasting objects. The ParcelHandler class is a base 
dispatcher class for handling parcels and gives agents the 
ability to communicate with one another using parcels. 
20 The Agent class consists of the following data members. 
A name (inherited from Typed) 
A type ID (inherited from Typed) 
A pointer to the session (inherited from ParcelHandler) 
25 A timeout (inherited from ParcelHandler) 

A task stack, which contains a linked list of task objects 

to be executed by the agent 
A pointer to the active gadget 
A pointer to the persistent agent 
30 The Agent class contains member functions for 

Handling tasks. This set includes functions for adding 
tasks, removing tasks, getting the next task from the 
task stack, performing a task, executing a task, and 
35 cancelling a task. 

Returning the active gadget and persistent agent. 
Handling each type of parcel contents (ParcelContents 
class). These member functions are inherited from the 
ParcelHandler class. 
40 Each subclass of Agent contains any objects belonging to 
that type of agent. For example, the subclass for the assistant 
contains the hand and band history, while the subclass for 
the courier agent contains the list of parcels to be delivered. 
In addition, each subclass of Agent contains methods for all 
45 actions carried out by the agent that are not tasks. 

All agents follow the same basic skeleton of operations. 
Specifically, an agent carries out the following operations in 
its main processing loop: 
Creates a session 
50 Creates an agent session within the session 
Enters the perform method to perform tasks 
Continues performing tasks until there are no more tasks 
or an exception is received 
55 Deletes its session 
Exits 

The Agent: :perform( ) method gets the next task on the 
task stack and performs it until there are no more tasks on 
the task stack. Utility agents, such as the cron agent and 

60 postmaster agent, however, do not use the task stack or tasks. 
Each utility agent knows how to handle a single type of 
parcel, so it provides its own perform( ) method that waits 
to receive a parcel and then services it. 
The main routine for a complex agent, such as the 

65 electronic assistant, has the basic operations just presented, 
but includes many additional operations, such as ones for 
initializing the memes in the database, looking up the slot of 
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the vmserver process, looking up the subscriber to which it 
belongs, getting the focus gadget, and so on. 
The ActiveGadget Class 

The ActiveGadget class is derived from the database (DB) 
class, although it is not actually stored in the database 
currently. The DB class is the primary subclass of Typed and 
it contains all of the objects stored in the database. The 
ActiveGadget class consists of the following data members: 

A pointer to the channel in the VM 

A pointer to the meme list 

A pointer to a menu 
The ActiveGadget class contains member functions for: 

Presenting a meme and collecting a response from a 
menu. 

Adding and removing capabilities. 

Training utterances. The set includes functions for starting 
training, collecting utterances, adding training, com- 
pleting training and aborting training. 
Hie ActiveGadget class defines operators for: 

Presenting information to the user («) 

Recording information from the user (») 

Gadgets store the "address" at which a user can be 
reached. There are the following types of gadgets: 

PhoneGadget, which describes the address of a 
telephone — the prefix, country code, city code, area 
code, number, and extension. 

AgentGadget, which describes the address of an 
assistant — a pointer to its subscriber's user object. 

ArdisGadget, which describes the address of an Ardis™ 
pager. 

SkyTel™ PagerGadget, which describes the address of a 
SkyTel™ pager — a pager ID and security ID. 

NumericPagerGadget, which describe the address of a 
numeric pager — a phone number. 

SocketGadget, which describes the address of a data 
network — a hostname, service, and service type. 
Theoretically, any agent can use any type of gadget. Certain 
agents, such as the courier agents, are specialized to deal 
with particular kinds of gadgets. For example, the system 
courier agent knows how to deal with the AgentGadget, 
while the Skytel™ courier agent and Numeric Pager courier 
agent know how to deal with the SkyTel™ PagerGadget and 
NumericPagerGadget respectively. 

To use a gadget to communicate with a user, an agent adds 
it to the session to obtain an active gadget. Then, the agent 
can send and receive information over the active gadget. 
The PersistAgent Class 

The PersistAgent class has no data members or member 
functions. Each subclass of PersistAgent provides data 
members for its agent's persistent data and member func- 
tions for accessing the data. For example, the persistent 
assistant contains the subscriber's box, while the persistent 
cron agent contains a list of cron events. 
The Task Class 

The Task class is derived from the both the DB class and 
the FieldParent class. The FieldParent class gives a task a 
way of knowing whether it is changed, completed, or 
cancelled. 

The Task class consists of the following data members: 
What has changed (inherited from FieldParent) 
Whether the task is complete (inherited from FieldParent) 
Whether the task has been cancelled (inherited from 
FieldParent) 
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A pointer to the agent 

A pointer to the active gadget 

A pointer to the session 
The Task class contains member functions for: 

Processing executing, and cancelling tasks. These func- 
tions are protected and accessed only by the Agent 
class. 

Returning and setting the active gadget. 
io Returning and setting the agent. 
Returning and setting the session. 
Each subclass of Task contains protected member functions 
for processing and executing that task. 
The Parcel Class 
15 The Parcel class is derived from the DB class and has the 
following data members: 
The address of the recipient 
The address of the sender 
20 A parcel ID 
A message ID 

A reference to the contents of the parcel 
The Parcel class has member functions for accessing data 
members of the class. It also has operators for: 
25 Filling an RPC output buffer with the parcel («). 

Removing a parcel from an RPC input buffer (»). 
The ParcelContents Class 

The ParcelContents class is derived from the DB class, 
30 and is an abstract class. The ParcelContents class contains 
no data members and a single member function for process- 
ing the contents of a parcel. 

Each subclass of ParcelContents contains data members 
appropriate for the parcel type, such as a reminder or even 
35 another parcel. Each subclass also has a member function for. 
processing its particular content. 
Setting up the Focus Gadget 

The focus gadget is a specific active gadget that an 
assistant must use to communicate with a user. In the case 
^ of a phone gadget, the focus gadget has speech-recognition 
resources associated with it. 

When the VM starts a session for an incoming call, it 
equips the session with a focus gadget, Focus, which is a 
data member of Session. For phone calls, this focus gadget 
45 is of type PhoneGadget. The assistant uses this focus gadget 
to communicate with the subscriber, contact, or non-contact 
that is calling. Before the assistant can use the focus gadget 
supplied with the session, it must take the following actions 
to set it up: 

50 1. Obtain the focus gadget with the get_iocus member 
function. 

2. Set any attributes needed on the focus gadget. 

3. Add any necessary capabilities to the focus gadget. 

4. Answer the call on the focus gadget. 

55 In the case where a session is started because of a parcel 
delivery, the VM does not create a focus gadget. In this 
situation, the assistant attempts to locate and notify the 
subscriber by making an outgoing call. Before the assistant' 
can communicate with the person being called, it must take 

60 the following actions to set up a focus gadget: 

1. Determine the gadget that corresponds to the subscrib- 
er's location. 

2. Obtain a focus gadget by adding the gadget to the 
65 session. 

3. Set any attributes needed on the focus gadget. 

4. Add any necessary capabilities to the focus gadget. 
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Once the focus gadget is set up, the assistant can commu- 
nicate with the user using memes and menus and manipulate 
the gadgets as necessary to make additional calls, place users 
on hold, and connect users. 
Obtaining Additional Active Gadgets 

Theoretically, agents can add any gadget to a session with 
the Session :add(Gadget) member function. When the agent 
adds the gadget, it receives an active gadget in return, and 
the focus is already set to that active gadget, making it the 
focus gadget. 

Each standard place (home, work, car, mobile, other) for 
each contact is represented by an addressable gadget object 
(e.g. a PhoneGadget object) that can be added to a session. 
For example, if a subscriber asks the assistant to place an 
outgoing phone call to a contact, the assistant obtains the 
phone gadget for the correct place (such as work), and adds 
it to the session. 

When the agent adds a gadget to a session, the VM obtains 
a phone line and dials the number specified in the gadget. 
The VM does not create an active gadget and give it to the 
agent until the call connects successfully. The criteria for 
successful connection is variable; it can mean the phone 
rang several times or that someone answered the phone. If 
the call fails to connect, the does not create an active gadget, 
and add operation fails for the agent. 
Setting Attributes and Capabilities 

Before the agent communicates over the active gadget, it 
must set any attributes and capabilities that it needs for 
communicating with the user. 

Attributes specify which media get presented in a meme 
and can be thought of as properties of MMUI objects. The 
language and type of prompts (brief, dialog, instructional) 
used by the assistant are examples of attributes. Most 
attributes are set as part of the user preferences mechanism 
in the assistant. When the assistant starts up, it sets attributes 
on the focus gadget for each user preference. Whenever the 
user changes a preference, the assistant sets the appropriate 
attribute. 

The VM currently provides the following set of capabili- 
ties: 

WflnterOutCap, which provides audio output 
WflnterlnCap, which provides audio output 
WfRecognizerCap, which provides speech and DTMF 

recognition 
WfFaxInCap, which provides fax input 
WfFaxOutCap, which provides fax output 
WfAllCaps, which provides all capabilities 
The assistant can set capabilities when it sets up the focus 
gadget or anytime during the session with the user. For 
example, if the user asks to fax some information, the agent 
can add the fax capability to the active gadget for carrying 
out that user request. 
Answering the Focus Gadget 

For incoming calls, the assistant must answer the focus 
gadget with the ActiveGadget::answer member function, 
which picks up the call and stops the ring tone. After the 
assistant answers the call, it can begin communicating with 
the user or callee (e.g. it could be a fax machine that called). 
Connecting Gadgets 

Agents can connect, disconnect, and reconnect gadgets as 
needed during a session. To connect gadgets that are located 
in separate sessions, agents use a separate object, the Con- 
ference object. Once a connection is established, agents can 
disconnect and reconnect gadgets to place users on hold and 
connect them back in. 
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Parcels 

Agents use parcels to communicate with one another. A 
parcel consists of a target (addresses), which can be a user 
or a session, and content (ParcelContents class). Parcels are 
delivered to the addressee's session. If the addressee is an 
agent that can have more than one session running, the 
parcel is always delivered to the master session. 

Table 5 describes the parcels that are used by agents. 
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Parcel/Contents Types 



ParcelContents Description 



Sending or 
Receiving Agents 



ConnectParcel 



CourierEvent 



CronEvcnt 



NewMailEvent 



PostalEvent 



RemindcrEvent 



Contains the caller 
requesting a connection and 
their name recording, a 
conference object, and the . 
response from the callee 
and the caller. The 
assistant uses this parcel 
to negotiate connections 
between incoming callers 
and subscribers. 
Contains a bond to the 
message to send, the list 
of addressees to which to 
send the message, and the 
number of retries left. 
The Postmaster agent uses 
this parcel to submit 
messages to courier agents 
for delivery. 
Contains a parcel to be 
delivered and the time at 
which it should be 
delivered. The assistant 
uses this parcel to submit 
reminders (or any other 
items that need delivery at 
a specific time) to the 
Cron agent. 
Contains a bond to a 
message. The assistant 
uses this parcel to submit 
messages to the Postmaster 
agent for delivery. 
The courier agents uses 
this parcel to indicate to 
the assistant that they 
have just delivered a 
message. 

Contains a bond to the 
message to send, the list 
of addresses to which to 
send the message, and the 
number of retries left. 
Courier agents use this 
parcel to request that the 
Postmaster retry delivery 
of a message. 
Contains a reminder. The 
assistant uses this parcel 
to package up a reminder. 



Postmaster 
agent, SkyTei " 
courier agent, 
system courier 
agent 



Assistant, Cron 
agent 



Assistant, 
Postmaster agent 



Assistant, 
system courier 
agent, SkyTei " 
courier agent, 
Secret agent 
Postmaster 
agent, SkyTei " 
Courier agent 



Assembling Parcels 

To assemble a parcel, the agent uses the Parcel constructor 
and specifies the ParcelContents class as an argument. The 
50 agent can then fill in the appropriate fields in the parcel's 
contents. 

Sending and Receiving Parcels 

To send a parcel, reply to a parcel or retrieve a parcel, the 
Session class provides the following member functions: 
65 send_parcel 

reply__to_parcel 
retrieve_parcel 
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When replying to a parcel, the VM automatically addresses 
the parcel to a specific session rather than a specific 
addressee. 

Passing the Same Parcel Back and Forth 

Agents can pass a parcel back and forth, amending its 
contents each time, as a means of negotiation. As an example 
of using parcels in this manner, consider how the assistant 
handles calls. When a subscriber calls a contact, there is no 
negotiation involved in connecting the telephone call. There 
is a three-part handshake involved in connecting a contact to 
a subscriber in the following two situations: 

An outside contact calls a subscriber; 

A contact that happens to be a subscriber calls another 
subscriber who is logged into the system 
Referring to FIGS. 23A-D, the three-part handshake 
involves the following steps: 

1. When a contact calls a subscriber, the electronic assis- 
tant 400 for the contact's session 402 sends a connect 
parcel 404 to the master session 406 (FIG. 23A). The 
parcel indicates that the caller is requesting a connec- 
tion to the subscriber and contains a reference to the 
contact (or a dummy contact if the caller is not a known 
contact). If there is no master session, the VM starts 
one. 

2. When the electronic assistant for the master session 406 
receives the connect parcel 404, it checks to see if it has 
the focus gadget 408, which indicates it already has a 
session running with the subscriber. If the electronic 
assistant does not have the focus gadget, it means that 
the session was started because it received a parcel, and 
it needs to try to locate and notify the subscriber in 
order to get the focus gadget. Once the electronic 
assistant has the focus gadget, it asks the subscriber 
whether or not to connect to the contact. If the sub- 
scriber says yes, the electronic assistant for the master 
session sends back the connect parcel 404 and indicates 
within it that the connection request has been accepted 
(FIG. 23B). 

3. When the electronic assistant for the contact's session 
receives the connect parcel, it sends it back with the 
indication to go ahead and make the connection (FIG. 
23Q. When the electronic assistant for the master 
session receives the connect parcel, it connects the two 
gadgets and moves the contact's active gadget into the 
master session (FIG. 23D). 

The way in which conference objects are used handling 
communications is illustrated in FIGS. 25A-D, which dia- 
gram the steps in creating a conference object for establish- 
ing a call from one subscriber to another subscriber. In this 
example, it is assumed that both subscribers are initially 
interacting only with their electronic assistants and thus are 
connected to idle conference objects. This is illustrated in 
session A by a focus gadget 401<j connected to an idle 
conference object 403a and in session B by a focus gadget 
4016 connected to an idle conference object 4036. 

Subscriber A initiates the call by instructing his electronic 
assistant to establish a connection to subscriber B. In 
response, the VM checks whether subscriber B is in session, 
i.e., whether a master session is running for subscriber B. If 
subscriber B is in session, the VM notifies subscriber A's 
electronic assistant that subscriber B is in a session on the 
system. This information is communicated to subscriber A, 
who then is given the opportunity to confirms his request for 
a connection. When subscriber A confirms his request for a 
connection, his electronic assistant sends a conference 
request 405 to subscriber B's electronic assistant and the 
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VM creates a conference object 407 for the requested 
connection (FIG. 24A). 

If subscriber B indicates to his electronic assistant that he 
will accept the requested connection, his electronic assistant 

5 sends an acknowledgment (ACK) back to the electronic 
assistant for subscriber A (FIG. 24B). In response to receiv- 
ing the ACK from subscriber B, subscriber A's electronic 
assistant constructs a conference parcel 409 identifying the 
conference object 407 which has been created for the 

10 connection, connects itself into the conference object 407, 
and sends the conference parcel 409 to subscriber B's 
electronic assistant (FIG. 24C). 

When the electronic assistant for subscriber B receives the 
conference parcel from subscriber A, it extracts the Identity 

15 of the conference object from the parcel, removes its active 
gadget from its idle conference object, and adds its active 
gadget to the newly received conference object 407 (FIG. 
24D). At this point, subscribers A and B are switched 
together and their electronic assistants let them know that 

20 they are connected by saying "Go ahead". 
The Functionality of the Electronic Assistant 

The set of commands that are accessible to the subscriber 
are presented in Exhibit A. Hie various tasks which the 
electronic assistant can execute implement the functionality 

25 associated with these commands. 

The following are more detailed descriptions of various 
functions that the electronic assistant performs for its sub- 
scriber using the mechanisms described above. The func- 
tions that are described include among others: (1) handling 

30 an incoming call to a subscriber, (2) creating a contact, (3) 
creating a reminder, and (4) notifying the subscriber of a 
reminder. 

Handling an Incoming Call 

First, we describe how the system handles an incoming 

35 call from a caller (i.e., Bill Bishop) to a subscriber (i.e., Jim 
Smith). The system's call handling functions are invoked 
when a caller places a call to the subscriber's phone number. 
The central office notifies the system of an incoming call by 
placing a ring signal on a Tl line into the system. The 

40 interface card that is monitoring that Tl line responds to the 
ring signal by picking up the line and acquiring a 4-digit 
sequence of touch tones identifying the extension that is 
being called. The interface card generates an event for the 
virtual machine. The virtual machine determines from the 

45 hardware what number was called and from that determines 
the identity of the subscriber being called. 

Upon receiving an indication of an incoming call, the VM 
starts up a session, i.e., a single thread of execution, that will 
be assigned to handling that call. When the VM server 

50 spawns the session, a handshake occurs between the session 
and the VM server to enable the two entities to communicate 
with each other. As part of the handshake, the server process 
passes to the session an ID of the subscriber for whom the 
session is being started up. Thus, the session is able to 

55 immediately establish a reference to a user object in the 
database that contains the stored information for the sub- 
scriber. 

In addition to receiving the ID of the user for which the 
session has been started, the VM also passes up the identity 

60 of the channel out of which it constructed an active gadget. 
The active gadget represents the communication path to the 
incoming caller. The active gadget represents the particular 
communications line that the call came in over, in this case 
a phone line. Until the capabilities are added, the active 

65 gadget simply represents a channel with no input or output 
capabilities. Therefore, to enable the system to play prompts, 
record incoming signals, and perform voice and DTMF 
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recognition on the channel, the assistant adds capabilities to caller the option to leave voice mail for the subscriber (step 
the active gadget. For example, the assistant adds 506). If the caller is a known contact, the electronic assistant 
interactive-out capability, interactive-in capability and rec- determines whether the known contact is, in fact, the sub- 
ogniuon capability. The capabilities that are added to the scriber for whom the electronic assistant is acting (step 508). 
active gadget map to ports which the VM attaches to the 5 If the electronic assistant determines that the caller is the 
channel. For example, interactive-out capability maps to an subscriber, the electronic assistant authenticates the sub- 
audio-out port for playing audio, the interactive-in capability scriber by asking for a password before allowing him to have 
maps to audio- in port for recording speech), and the recog- access to the system commands (step 510). Once the identity 
nition capabilities map to speech recognition and DTMF of the subscriber is verified, the electronic assistant notifies 
ports. 10 the subscriber of all new messages that have been stored 

Once the capabilities are set up, the electronic assistant since he last read them and it starts the command task which 

invokes an answer call task to handle the incoming call. gives the subscriber access to the set of commands for 

Referring to FIGS. 24A-B, the answer call task engages in controlling the electronic assistant (step 512). 

a dialogue with the caller to collect certain information If the caller is not the subscriber, the electronic assistant 

which will be necessary to beginning execution of the task. 15 checks the subscriber's contact list to determine the priority 

In general terms, the answer call task initially has two fields of the caller (step 516). Recall that the subscriber's contact 

of information that must be filled in: WHO is calling and list includes the subscriber's designation of each contact's 

WHAT would the caller like the electronic assistant to do. priority (i.e., normal or high). The priority indicates the 

When an incoming call arrives for a given subscriber importance of calls that are received from the caller, 

there are a number of ways in which the assistant might 20 While checking the priority of the caller, the electronic 

handle the call, depending on the preferences which the assistant also checks the data structure for the contact to 

subscriber has previously selected. The assistant might determine whether there are any voice messages left for the 

directly forward the call to a telephone on the subscriber's caller by the subscriber (step 518). If there are voice 

desk phone, it might simply offer to take a message from the messages, the electronic assistant invokes a deliver message 

mroming caller, or it might attempt to locate the subscriber 25 task which asks the caller whether or not he wishes to hear 

and offer to connect him to incoming call once he is located. the voice messages left by the subscriber (step 520). 

In handling the call, the answer call task first checks the The deliver message task performs the following 

subscriber's status to determine which preference he has sequence of events. First, the electronic assistant sends a 

selected (step 500). meme to the channel instructing it to say to the caller "I have 

In the described embodiment, it is assumed that the caller 30 a message, would you like to receive it?". It also passes a 

wishes to have a connection established with the subscriber, pointer to the stored voice message in its message database, 

thus the caller is not offered any options for responding to The caller responds by pressing the appropriate buttons on 

the second question. However, in alternative embodiments, the touch-tone phone. If the caller presses "9" for "yes**, the 

the answer call task could be modified to offer the caller the audio-out port retrieves and plays the appropriate media 

option to leave a message, to specify the priority of the call, 35 representation of the stored message. After the message is 

or to send a page, among other things. played, the electronic assistant checks whether there are any 

Also note that if ANI (automatic number identification) is other unplayed messages, and if there are, it goes through the 

supported, the interface card could have stripped off the ANI same sequence of operations to present the other messages 

signal that it received from the central office and supplied it to the caller. 

to the electronic assistant for the session. If that number 40 Note that the deliver-message task of the described 

mapped to a contact in the subscriber's contact list, the embodiment does not perform any verification of the caller's 

electronic assistant could supply the "Who?" entry auto- identity. It may, however, be desirable to do so. This can 

matically without engaging in dialog with the caller. Even in easily be done by modifying the task to request a verification 

that case, however, it may still be desirable to engage in a such as a password. 

dialog with the caller to confirm the caller's identity. 45 After the deliver message task is complete or if there are 

After the answer call task determines that the subscriber no voice messages for the caller, the electronic assistant 

wants calls screened, the answer call task attempts to iden- checks the availability of the subscriber (step 522). The 

tify the caller (step 504). The electronic assistant instruction subscriber has the ability to designate his availability 

passes an object (i.e., a meme object) to the active gadget through the "I Will Be" command (described elsewhere), 

instructing it to say "Please say your name." It also passes 50 This information is stored in the database as part of the 

the subscriber's contact menu to the gadget so that the subscriber's user object. 

caller's reply can be matched against the subscriber's con- After identifying the subscriber's availability, the elec- 

tacts to determine whether or not the caller is a known tronic assistant determines whether or not to attempt to 

contact. The electronic assistant asks the gadget to return the establish a connection to the subscriber (step 524). In the 

identified name. 55 described embodiment, the determination is based on 

The gadget presents the "Please say your name" meme to whether the caller has high enough priority to meet the 
the channel. The channel presents the meme to the collection availability criteria established by the subscriber. For 
of attached ports and the appropriate port plays the meme by example, if the caller is identified in the database as a normal 
retrieving the stored audio version from the systems data- priority contact and the subscriber has indicated his avail- 
base. 60 ability to be "taking important calls", then the electronic 

Upon receiving the caller's spoken response, the channel assistant does not attempt to establish a connection to the 

attempts to match the response against responses stored for subscriber for this particular call. On the other hand, if the 

known contacts in the contact menu. The result which is contact's priority is designated as high priority, then the 

passed up to the electronic assistant will either be the electronic assistant performs the operations necessary to 

identity of a known contact or an indication that its an 65 establish a connection with the subscriber, 

unknown contact. If it is an unknown contact, the answer The decision algorithm in the described embodiment is a 

call task invokes a take a message task which offers the very simple one. Alternative embodiments might use much 



09/02/2003, EAST Version: 1.04.0000 



6,047,053 

35 36 

more elaborate algorithms to filter the caller's request for a known by the electronic assistant for the second subscriber, 

connection to the subscriber. For example, the decision Once this is done, the schedule of the first subscriber will 

could take into account other information about the caller track the schedule for the second subscriber who the first 

such as his job description, or the identity of the company subscriber is with. 

from which he is calling. Also, it would be a simple matter 5 The locate-and-notify task checks the subscriber's over- 

to let the priority of the caller change to reflect the number ride schedule to determine whether the subscriber has an 

of calls that the subscriber has initiated to that caller within override for that time. If no override exists, the electronic 

some preselected period of time. A contact that the sub- assistant checks the default schedule. If the default schedule 

scriber calls frequently would have a priority that is higher identifies a location for that time, the electronic assistant can 

than a contact to whom the subscriber rarely places a call. find the aumber for that location hom ^ mfor . 

If the decision 1S made to not attempt a connection to the matioQ stored in lhe subscriber - s user object For example, 

subscriber the electronic assistant invokes a take message tf ^ schcdule indicates ^ ^ subscriber is Khed _ 

task (step 526). In general, the take message task reports to , , , , A , , t , 4 . . . . , , - 4U 

the caller that the subscriber is not available at that moment ulcd l ° bc at b ° m ? a < th f m ° ment > th * 98Sastan } 1°°**™ * e 

and asks the caller whether or not he wishes to leave voice uscr °^ ect to find the ! cle P h ° ne number associated with the 

mail for the subscriber. After the take-message task is « home location, assuming of course that the subscriber has 

complete, the electronic assistant performs whatever opera- provided one. 

tions are necessary to disconnect from that channel and 0oce the appropriate communications device and its 

terminate the session for that caller. This includes, for number (or address) is identified, the electronic assistant 

example, freeing up the resources that were required to adds a gadget representing that device to the session (step 

support the connection from a caller by stripping off the 20 604). If the communications device is the subscriber's home 

capabilities that are associated with the active gadget and phone, the gadget that is added to the session is a phone 

deallocating the audio-in, audio-out, and recognition ports gadget that contains the phone number for the subscriber's 

from the channel so that they may be used by other sessions. home phone. 

If the subscriber has indicated that he is available to In response to receiving a gadget from the master session, 

receive calls from that contact, the answer call task performs 25 the VM allocates a channel to that gadget. Since the gadget 

the steps that are necessary to establish the connection (step is a telephone, the VM causes the appropriate interface card 

528). To further illustrate the operation of parcel mechanism, to allocate a phone line channel connecting to a Tl line. The 

the exchange of parcels that takes place to assist in estab- VM also passes the telephone number to the interface card 

lishing the connection will also be described. and the interface card dials that number. If the called number 

The answer call task in the caller's session sends a 30 is busy, the channel reports back to the master session that 

connect parcel addressed to the subscriber. The connect the attempt to connect failed. If the phone begins ringing, the 

parcel requests that a connection be established with the channel returns an active focus gadget to the master session, 

subscriber and it identifies the caller's session. The VM The active gadget represents an actual connection to the 

handles the delivery of the parcel. If the VM determines that subscriber's home phone. 

no master session currently exists, it spawns a new session 35 Upon receiving the active gadget, the electronic assistant 

which it designates as the master session and then delivers immediately assigns capabilities to the active gadget to 

the connect parcel to that master session. enable it to communicate through the gadget and waits for 

Once the master session exists, it immediately checks somebody to pick up the phone (step 606). If a party 
whether any parcels have been delivered to it. At this point answers, the electronic assistant determines whether the user 
it will discover the connect parcel that was sent by the 40 is available (step 608). It does this through a dialog during 
caller's session. In response to the connect parcel, the master which the answering party can indicate whether the sub- 
session checks whether it has the focus gadget. If it does not, scriber is there to receive the call. In the described 
it invokes a locate-and-notify task the purpose of which is to embodiment, the electronic assistant causes the audio-out 
attempt to establish a connection with the subscriber. port to play the previously described stored message: 

Referring to FIG. 26, the locate-and-notify task first 45 Hello, I'm trying to reach Jim Smith. If he is available, 

attempts to determine where the subscriber can be reached press the 9 key. If he is not available, press the 6 key 

(step 600). In the described embodiment, the assistant or hang up. 

checks the subscriber's user object for stored schedules. It might be desirable to tailor the dialog to take into 

There are two schedule data structures which are used to list account the location of the phone that was just called. For 

the schedule for a user: a default schedule and an override 50 example, one might wish to designate phones as being in 

schedule. The default schedule indicates where the sub- friendly territory, questionable territory, or hostile territory, 

scriber can typically be found throughout a normal day or A phone that is in friendly territory might be any phone for 

throughout the days of a normal week. The override sched- which it is likely that the person answering it is familiar with 

ule that the subscriber can generate through the "I-Will-Be" interacting with the electronic assistant. A phone that in 

or "Create-an-Itinerary" commands, overrides the default 55 questionable territory might be is any phone for which there 

schedule for the relevant periods of time. For example, the is a 50/50 chance that the answering party is not experienced 

subscriber may usually be in his office on Mondays and his in interacting with the electronic assistant Phones in hostile 

default schedule will reflect this. However, on a particular territory might be any phone for which there is a reasonably 

Monday he may have to drive elsewhere to visit a client. He high likelihood that the answering party is not experienced 

can generate an override schedule to reflect this change from 60 in interacting with the electronic assistant. By identifying 

normal routine. The override schedule would indicate that he the connection devices in this manner, an appropriate dialog 

is reachable through his mobile phone for the period of time for interacting with the answering party can be selected. For 

that he is expected to be traveling by car and that he is the inexperienced user, the electronic assistant might use a 

thereafter reachable at the client's business location for verbose dialog which explains how to interact with it more 

another period of time. 65 fully. Whereas, for the experienced user, it might use a terse 

The subscriber could also set his location as being with a dialog which assumes that the person knows how to respond 

second subscriber whose schedule and whereabouts are without being told. 
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If the answering party indicates that the subscriber is not 
available, this is reported to the master session which then 
sends a reply parcel back to the caller's session indicating 
that the connection attempt failed (step 610). The master 
session then goes through the sequence of steps that is 
necessary to terminate the session. This includes stripping 
the capabilities from the active gadget. 

If the answering party indicates that the subscriber is 
available, then the electronic assistant announces the iden- 
tity of the caller to the subscriber (step 612): 

There is a call from Bill Bishop. Do you want to take the 
call? Indicate Yes by pressing the 9 key, indicate No by 
pressing the 6 key. 
It then asks the subscriber whether he wishes to accept the 
call (step 614). 

If the subscriber declines the call, the locate-and-notify 
task sends a failed connection reply parcel back to the 
caller's session (step 616). In response to the fail reply, the 
caller's session informs the caller that the subscriber could 
not be found and it offers the caller the option of leaving a 
recorded voice message for the subscriber. If the caller 
accepts the message option, the electronic assistant records 
the caller's voice message and stores it for the subscriber to 
play back at some later time. 

While the connection to the subscriber still exists, the 
locate-and-notify task asks the subscriber whether there is 
anything it can do for the subscriber at this time (step 618). 
If the subscriber responds by saying no or pressing the "6" 
key, the locate-and-notify task says "Goodbye" to the 
subscriber, sends a hang-up command to the channel, and 
then terminates. On the other hand, if the subscriber indi- 
cates that he wants to access his electronic assistant's 
commands, he must first do additional authentication, then 
the electronic assistant starts a command task which enables 
the subscriber to access those capabilities either through 
voice or DTMF commands (step 620). The commands 
identified in Exhibit A are then available to the subscriber. 

If the subscriber accepts the call, the locate-and-notify 
task sends a connect reply parcel to the caller's session. 
Referring to FIG. 24B, in response to receiving the connect 
reply parcel, the caller's session removes from its active 
gadget some of the capabilities that will no longer be 
necessary (e.g. interactive-in, interactive -out, and recogni- 
tion capabilities) (step 536) and sends the gadget to the 
subscriber using the parcel mechanism (step 540). Upon 
receiving the active gadget from the caller's session, the 
electronic assistant for the master session adds that active 
gadget to its session and connects both the active gadget and 
the focus gadget together thereby enabling the subscriber 
and the caller to communicate with each other (step 622 in 
FIG. 26). The interface cards perform the switching function 
that connects the two gadgets together by allowing each 
gadget to listen to the output of the other channel (i.e., by 
allowing each channel to have access to the appropriate time 
slot on the MVIP bus line that carries the other party's 
signal). 

When the connection is established between the caller and 
the subscriber, the electronic assistant switches into a back- 
ground mode in which it monitors the subscriber's output for 
a command (i.e., "Wildfire") that will call it back into its 
foreground mode (step 624). While the electronic assistant is 
in the background, it only responds to the single command 
which calls it back into the foreground and it ignores all of 
the other commands in its command set. 

Recall that it was assumed in the above discussion that a 
master session was not running when the VM wanted to 
deliver the request connect parcel from the caller's session. 
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If, however, there is a master session running, the VM 
delivers the parcel to that master session. 

When an already existing master session receives a 
request to connect parcel from another caller's session, it is 

5 stored in a queue. The master session repeatedly checks this 
queue for received parcels. When it detects the presence of 
a new parcel, it acts upon it immediately. In the case of the 
request to connect parcel, the master session initiates a 
notify task (see FIG. 27). The notify task checks whether the 

10 subscriber is interacting with the master session through a 
phone channel (step 630). If the subscriber is interacting 
with the session through communication channel other than 
a phone channel, it sends a notification to the subscriber 
which identifies the caller and notifies the subscriber that he 

15 has just received a call from that individual (step 632). The 
master session then sends a reply to the request to connect 
parcel back to the caller session indicating that the attempt 
to connect failed (step 634). 

If the caller is communicating with the master session 

20 through a phone channel, the notify task first determines 
whether the subscriber has indicated that he is willing to 
accept an interruption to his call (step 636). For example, 
when the subscriber connected to the call he could have used 
a "Hold All Calls" command to indicate his preference to not 

25 be interrupted. The "Hold All Calls" command temporarily 
sets the subscriber's status as unavailable during a call being 
handled by the master session. 

If the subscriber indicated that he did not want to be 
interrupted, the notify task sends a non-interactive notifica- 

30 tion to the subscriber of the caller's attempt to reach him and 
then reports the caller's session that the connect attempt 
failed (step 634). 

If the subscriber did not set his status to block 
interruptions, the notify task notifies the subscriber of the 

35 caller on hold (step 638). The system has a form of call 
waiting in which it first plays a short tone that can be heard 
by both the subscriber and the party he is talking to and then 
it plays the name of the incaller, using the caller's spoken 
self identification. The interruption is handled so as to 

40 prevent the caller with whom the subscriber might be talking 
at that moment from hearing the message (referred to 
generally as the smart call waiting feature). On some phones 
this is done by placing the caller on hold for a short period 
of time while the announcement is made. On other phones, 

45 it is possible to send the message to the earpiece on the 
subscribers phone without the caller hearing it. 

The electronic assistant then gives the subscriber the 
option to reject the call or to accept the call. Current valid 
commands are Take-A-Message-and Tll-Take-lL If the sub- 

50 scriber rejects the call or does not respond within a prese- 
lected period of time, the electronic assistant sends a fail 
reply to the caller's session (step 642). In response to the fail 
reply, the caller's session may simply inform the caller that 
the subscriber could not be found or it may offer to take a 

55 message from the caller. If the caller accepts the message 
option, the electronic assistant records the caller's voice 
message and store it for the subscriber to play back at some 
later time. 

If the subscriber accepts the new call, the electronic 
60 assistant merges the caller's gadget into the master session 
(step 644) and places the first caller on hold. 

While the subscriber is connected to the new caller, if yet 
another call comes, the electronic assistant handles the new 
call in the same manner. Thus, it is possible for more than 
65 one caller to be on hold at the same time. 

To switch back to the first caller, the subscriber issues a 
"Press-the-hold-button" command. If there is only one call 



09/02/2003, EAST Version: 1.04.0000 



6,047 : 

39 

waiting, the electronic assistant puts the second caller on 
hold and switches to the first caller. Thus, by using the 
"Press-the-hold -button" the subscriber can toggle back and 
forth between the two parties. However, if there is more than 
one call waiting, the electronic assistant responds to the 5 
"Press-the-hold -button" by placing the caller with whom the 
subscriber is presently speaking on hold and saying: "Shall 
I connect you with <caller's name>?" If the subscriber 
responds by saying "yes", the electronic assistant connects 
the subscriber to the identified caller. If the subscriber 
responds with "No", the electronic assistant again says 
"Shall I connect you with <next caller's name>? In this 
manner, the electronic assistant proceeds cycles the group of 
parties that are on hold until the subscriber indicates that he 
wishes to establish a connect ion with one of the parties. 
Create Contact 15 

Among the other commands available to the subscriber 
are certain non-message object creation commands. These 
are used to create contacts which are added to the subscrib- 
ers contact list, and to create call and recorded' reminders 
both of which are held by the cron agent for delivery to the 20 
subscriber at the appropriate time. Through a phone con- 
versation the subscriber invokes these commands by saying: 
" Create- A-Contact", "Remind-Me-To-Call" and "Remind- 
Me". An example dialogue which results from invoking the 
"Create-A-Contact" command is shown in FIG. 28. The 25 
mechanisms for implementing this dialog through the chan- 
nel (and other dialogues which are described below) are the 
same as were described earlier and will not be repeated in 
the following discussions. 

In response to receiving a "Create-A-Contact" command, 30 
the electronic assistant asks the subscriber "What kind — 
person, place or group?". The subscriber responds by iden- 
tifying one of these types. In the illustrated dialogue of FIG. 
28, the subscriber responds by saying "Person". The elec- 
tronic assistant then asks for the person's name. Since the 35 
electronic assistant will use the subscriber's vocalization of 
the contact's name, it asks the subscriber to repeat the 
contact's name a second time to improve the quality of the 
stored vocalization. 

After the subscriber has identified the contact's name, the 40 
electronic assistant requires the subscriber to add a phone 
number by asking: "Which phone number?". In response, 
the subscriber can identify home, work, mobile or other. In 
the illustrated example, the subscriber responds by saying 
* "Work". The electronic assistant then asks for the phone 45 
number and the subscriber enters the number followed by a 
# key. The electronic assistant acknowledges receipt of the 
number by saying "Got it. Now what?" 

If the subscriber wants to enter additional information for 
that contact, he uses the "Update-It" command. The elec- 50 
tronic assistant responds by asking "What?". To this the 
subscriber responds by identifying the particular item of 
information which he wishes to add or update. If the 
subscriber says "Priority", the electronic assistant asks 
"Normal or high". After the subscriber selects one of the 55 
available priorities, the electronic assistant indicates it is 
ready for another command by saying "Got it". If the 
subscriber wishes to add or modify further information for 
that contact, he again uses the "Update-It" command fol- 
lowed by an indication of what information is to be added or 60 
changed. In the illustrated example, the next item selected 
for update by the subscriber is the spelling of the contact's 
name. In response, the electronic assistant notifies subscriber 
that it is ready to accept the new information by saying 
"Begin spelling now". 65 

After the spelling has been entered, the electronic assis- 
tant indicates its readiness for the next command from the 
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subscriber. If no command is forthcoming within a prede- 
termined period of time, the electronic assistant prompts the 
subscriber for a command. If the subscriber indicates that he 
has no further additions or changes at this time, the elec- 
tronic assistant indicates that the "Create-A-Contact" is over 
by saying the "Done". 

The subscriber can also create new contacts by copying 
them from the phone book. Phone book entries, however, do 
not have voice identifications associated with them. Thus, if 
the subscriber copies an entry from the phone book, he must 
add a voice identification to make it part of his contact list. 

Finally, a contact can be created as a result of receiving a 
message with an enclosed contact. When reviewing a mes- 
sage which contains a contact, the assistant asks the sub- 
scriber if he wishes to transfer the contact out of the message 
and into the subscriber's contact list. If the subscriber 
responds by saying "yes", then an procedure that is similar 
to the one for saving phone book listings is executed. 

It should be noted that the system is capable of capturing 
phone numbers under other circumstances and later use this 
captured information to assist the subscriber. For example, 
when the assistant responds to a call and offers the caller the 
option of leaving a message for the subscriber, it first asks 
the caller to enter his phone number. After the phone number 
has been entered, the electronic assistant then records the 
caller's voice message. Also when the subscriber dials 
another number through the electronic assistant, that dialed 
phone number is remembered. This remembered informa- 
tion can later be used in response to a "Give-Them-A-Call" 
command or to create a contact for the person or place that 
is represented by the remembered phone number. 
Remind-Me 

As indicated above, the subscriber can use the commands 
"Remind-Me" and "Remind- Me -To-Call" to create remind- 
ers. The dialogs associated with these tasks are shown in 
FIGS. 29 and 30, respectively. 

To create a reminder, the subscriber issues the "Remind- 
Me" command. This causes the assistant to invoke the 
reminder task with an attribute to indicate the creation of a 
recorded reminder. At this point the assistant prompts for 
what the user would like to be reminded about. At that point 
it starts recording as a message whatever the subscriber says 
into the phone, to be played when the reminder comes due. 
The subscriber indicates the end of the message by pressing 
the "#" key and the electronic assistant stores the recorded 
message in its database. 

After the recording is complete, the electronic assistant 
engages in further dialogue with the subscriber to establish 
a time at which the reminder is to be delivered. The 
electronic assistant asks "When?". The subscriber responds 
using one of several conventions that are available in the 
system for specifying a time. The conventions are shown in 
Table III. In the described example, the presenter responds 
by saying "Today". The electronic assistant then asks the 
subscriber for a time. Using touch tone signals DTMF, the 
subscriber enters a specific time (e.g. 530 for 5:30). The 
electronic assistant responds by restating the time which the 
subscriber has entered and asks for a confirmation from the 
subscriber. If the subscriber confirms the time, the electronic 
assistant indicates that the dialogue has concluded success- 
fully by saying "Got it, I'll set my watch alarm.". If the 
subscriber does not confirm the time (either because the 
electronic assistant made a recognition error or because he 
changed his mind), the electronic assistant then repeats the 
above described sequence of operations to obtain a new 
time. 

If the subscriber issues the "Remind-me-to-call" com- 
mand (see FIG. 30), the create-reminder task asks the 
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subscriber "Call whom?*'. The subscriber's response "John scheduled reminder stack (step 710). If an alternative means 

Smith" is recognized by the voice recognition capabilities of for notifying the subscriber does exist, the electronic assis- 

the ASR card and matched against utterances stored with the tant sends a notification to the alternative gadget (step 708) 

subscriber's contact list that identify the subscriber's con- before placing it on the non-scheduled reminder stack, 

tacts. If a match is found (i.e., if a contact from the 5 If the locate-and-notify task establishes a connection with 

subscriber's contact list is identified), a pointer to that me subscriber, the reminder is played back to the subscriber 

contact is saved as part of this call reminder. ( ste P 720 ) Aftcr that ' the electronic assistant asks the 

The electronic assistant then engages in the previously subscriber if he wants to reschedule the reminder (step 722) 

described dialogue with the subscriber to obtain a delivery lf u he ***** * reschedule it, a reschedules task is executed 

4 . e .i . . a n, t i j ■• . u l „ that enables the subscriber to change the delivery time for 

time for the reminder. After the delivery time has been 10 iL • j / * ^ J jC * • j • 

. 4 , . t • . * i .t_ t w ~. , j the reminder (step 724). The modified reminder is then sent 

entered, the electronic : assis ant asks the subscriber: Should back tQ ^ ^ » / fof ^ ^ {hc rescheduled time< 

I know the topic for the call? . If the subscriber responds in ^ hm6{Q reminder task then ^ ^ ^^.^ loop 

the affirmative, the electronic assistant says "Recording . . . mnoing which enaWes ±c Sllbscriber to access me fa u set of 

" to notify the subscriber that he may begin recording. As commail ds (step 726). Additionally, at any time during a 

before, the subscriber ends the recording by pressing the # is se^io^ a subscriber can "Find" and reschedule pending 

key. In response, the electronic assistant confirms that the reminders using the "Update-it" task. (See Exhibit A), 

command has been received by saying "Got it, I'll set my if the subscriber elects not to reschedule the reminder, the 

watch alarm". reminder is placed on a non-scheduled reminder stack (step 

The electronic assistant sends the completed reminder to 725) and the handle reminder task starts up the command 

the Cron agent via the parcel mechanism. First, it packages 20 task loop (step 724). If the call reminder identifies a contact, 

the reminder in a parcel that is addressed to the subscriber the subscriber can use the "Call Them" command at this 

and includes its desired delivery time. It then places that point. Since the call reminder includes a pointer to the 

parcel in a second parcel that is addressed to the Cron agent. contact in the subscriber's contact list, the electronic assis- 

The VM delivers the outer parcel to the Cron agent. Upon torn interprets the "Give-Tbem-A-CalT command as though 

receipt, the Cron agent opens the parcel and pulls out the 25 il was equivalent to a "Call" command where the contact is 

parcel that is inside; it checks the delivery time for that ^ one ™ hand * ^ electronic assistant uses the gadget 

parcel and places it in a time ordered queue. The Cron agent identified in the contact's object to place the call 

keeps track of the delivery time of the top parcel. l f l°cate-and-notify task reaches the subscriber ^s 

Handle Reminder Task ^get but the connection to the subscriber is not accepted, 

At4 . , 4 . ^ , . . the electronic assistant places the reminder in the active 

At the delivery time, the Cron agent wakes up and sends 30 rcm j nc j er s t ac k ( s i c 703) 

me remmder parcel to the indicated address. As described rC othe'/tX that'are implemented by the system are best 

earlier, the VM handles the delivery of the reminder parcel. describe d through examples of the commands and the dia- 

If there is no master session running, the VM starts up a fogs mat the system supp orts when interacting with the 

session and delivers the parcel to that new session. If there subscriber. A summary of the commands is presented in 

is a master session running, the VM delivers the parcel to the 35 Exhibit A at the end of this specification, 

master session. Find Command 

Referring to FIG. 31, when the session that receives the Referring to FIG. 32, the subscriber can retrieve and 

parcel detects that it has received a reminder parcel, it starts review access various items of information that are stored 

up a handle reminder task. The handle reminder task checks for him by using the Find command. When the subscriber 

whether the current session is a master session (i.e. whether 40 says "Find" while the electronic assistant is running the 

the session has the focus gadget) (step 700). If the session is command task loop, the electronic assistant replies by ask- 

communicating with the subscriber, the electronic assistant ing "Find what?". At this point the subscriber can identify 

notifies the subscriber of the reminder using the existing one of seven different items, namely, contact, all-the- 

focus gadget (step 702). The method used for notifying the contacts, phonebook-listings, messages, new messages, 

subscriber depends upon the channel through which the 45 messages from, saved messages, reminders, and trash. The 

subscriber is connected to the session. If the subscriber is operation of the Find command will be illustrated using the 

connected through a phone channel, the electronic assistant contacts as the item which the subscriber wishes to manipu- 

briefly interrupts his call, reports to him that a reminder has late. 

come due. After notifying the subscriber of the reminder, the When the subscriber is asked for what it is that he wants 

electronic assistant places the reminder in an active reminder 50 to find, he replies by stating "All-The-Contacts". The elec- 

list in the box, where the subscriber can review it during the tronic assistant then accesses the subscriber's contact list 

current or any subsequent sessions (step 703). The user can and reports to the subscriber how many contacts are on the 

then invoke a "Find-Active-Reminders" command to deal list. Then the electronic assistant waits for a command from 

with the recently received reminder. the subscriber. The subscriber can search the contact list for 

If the subscriber is not connected to the session, the 55 a given name by issuing the Find "Contact" command. In 

handle reminder task initiates a locate-and-notify task. The response, the electronic assistant asks "Contact name?". The 

locate-and-notify task operates in the manner previously subscriber responds with a name, e.g. John Smith. The 

described except that instead of notifying the subscriber of electronic assistant acknowledges by replying "Contact 

a caller's attempt to reach him, it notifies the subscriber of name, John Smith" to indicate it has the user object for that 

the reminder. If the locate-and-notify task is unable to 60 contact in its hand. The subscriber can then instruct the 

establish a connection with the subscriber, it sends a failed- electronic assistant to: (1) describe it by issuing the describe 

to-connect reply to the handle reminder task. In response, command; (2) update it by issuing update command; (3) 

the handle reminder task checks whether the subscriber can discard it by issuing the "Throw-It-Away" command; or (4) 

be reached through a non-interactive means by, for example, call them using the "Give-Them-A-Call" command, 

checking for a pager assigned to the subscriber. If no 65 If the subscriber issues the describe command, the elec- 

altemative means for notifying the subscriber exists, the tronic assistant replies by reporting to the subscriber the 

electronic assistant places the reminder on the non- information that is stored for that contact. 
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If the subscriber issues the update command, the elec- for how many hours to which the subscriber responds with 

tronic assistant initiates a dialog similar to that previously a number. After the subscriber has indicated the time, the 

described with the Create command which enables the electronic assistant acknowledges his response by saying 

subscriber to modify or add to the information stored for that "Done". 

contact. 5 The "Create-An-Itinerary" command, a sample dialog of 

The subscriber can instruct the electronic assistant to dial which ^ snown ^ piG. 34, operates in a similar manner 

a number by saying "Call" which causes the electronic except tnat it allows the subscriber to use voice commands 

assistant to invoke the call task. The call task asks the to build as , a override schedule as is desired. In other 

subscriber to provide the number, which may be provided WOfd after the fifSt eQt has been the electronic 

either as a reference to . a i contact, by entering a specific 1Q ^ ^ ^ , u be „ fa ^ ^ gub _ 

number or by saying this-one , referring to a contact .^^^^^^^^^^^^ 

message or call reminder being pointed to; then, it dials the . , . , ( _ , , , , „ 

i_ A lL • - f. - *u i 4 • the command routine by sayine Back on schedule, 

number. As soon as the ringing begins, the electronic ass is- ... " , y y B 

tant establishes a connection is between the subscriber's line Calling Commands m 

and the outgoing call line so that the subscriber can complete A subscriber can ask his electronic assistant to place a 

the call. When that connection is made, the electronic 15 P hone cal1 m onc of mree wavs - If me to be called 15 

assistant automatically switches into its background mode in a known contact, the subscriber can identify the contact's 

which it will only respond to a particular command (e.g. name and the electronic assistant will obtain the information 

"Wildfire") which causes it to switch back into the fore- necessary to place the call from the subscriber's contact list. 

ground. In its background mode, the electronic assistant If the party is not a known contact, the subscriber can 

disables its full command set so that the utterance of the 20 identify the party and provide, either by voice or by DTMF, 

command words during the course of the conversation will a phone number to call. The third way is by relying on 

not unintentionally invoke a command task. information available from context. That is, if the electronic 

Anytime the electronic assistant is running its command assistant has just retrieved a voice mail message for the 

task which give the subscriber full access to its command subscriber, the subscriber can say "Give-Them-A-CalT and 

set, the subscriber can instruct the electronic assistant to go 25 me electronic assistant will call the party that left the voice 

into its background mode. He does this by saying the "That mail message> Additionally you could say "Call" "This- 

wffl be all for now". In response the electronic assistant 0nc „ ^ is possMe bccausc the ^ored message identifies 

replies, "Say Wildfire when you need me and switches into ^ caJlef either as a CQntact or b a telephoDe that the caller 

its background mode. When the subscriber needs to access wag to i eave 

the full command set of the electronic assistant, he says 3Q ^ stored objecte for the ^ a mter 

Wildfire \ In response the electronic assistant moves back to a note ( a wice } which , he subscriber can 

into its foreground mode and replies Here I am , confirm- aK; and attach , 0 ^ when me subscriber 

ing that the electronic assistant is again fully active and that his electronic to caU the me 

the subscnber may now access the full set of commands that eIectronic p , ays the ^ond message to the sub . 

are supported by the electronic assistant. 35 ^ it is attempting to establish the connection with 

Generating an Override Schedule me contact. The note might include information about the 

As indicated above, the I-WiU-Be and Create-An- which ^ subscriber wishes to te remindcd of 

Itinerary Commands enab e the subscriber to generate an whenevcr he calls hat For 6 le> he m ^ to 

override schedule. A typical dialog for each of these com. ^ ^ ^ of c0ntact . 5 w mat he M address 

mands is shown in FIGS. 33 and 34, respectively .The .dialog ^ hef b name tf ^ answels the hone 

for each of these commands is very similar. The I-WiU-Be Request Connection 

command is used for generating a single modification to the u ^ ibk for a subscriber lo MeT l0 hi s ^lanl that 

subscriber s schedule; the Create-An-Itinerary command fae WQuld ^ tQ ^ ^ mbacdba actu . 

is used for -generating more complex override schedule aU a ^ to faer ^ fa refefred to ^ re ting a 

When the subscriber utters I-Will-Be the electronic 45 connection ^ anome r user. One mechanism for accom- 

assistant responds by asking Doing What? In the ^ ^ ^ fof ^ subscribcr to explicitly a 

desenbed embodiment, the subscnber may respond in one of conncction . ^ causcs mc subscriber's assistant to inform 

the four following way: tfae clcclronic assistant to whom the request is being made 

Taking Calls that a connection at some point is desired. When this request 

Only Taking Important Calls 50 is received, the receiving assistant holds onto the request 

Unavailable until it is communicating with its subscriber. During the next 

Running on Schedule session with the subscriber, the receiving assistant informs 

"Taking Calls" indicates to the electronic assistant that all its subscriber of the requested connection and the availabil- 

calls should be forwarded to the subscriber; "Only Taking ity of the party requesting the connection. The subscriber 

Important Calls" indicates that only calls from contacts that 55 can then decide to connect with the other party or ignore the 

are designated as high priority should be forwarded; request. 

"Unavailable" indicates that no calls should be forwarded; Feature Phone 

and "Running on Schedule" indicates that the override The feature phone is an object within the system that 
schedule is being cancelled in favor of the default schedule. builds smart-phone like functionality on top of the confer- 
In the illustrated example, the subscriber reponds by 60 ence object. The feature phone allows the user to manage 
saying "Taking Calls." Next, the electronic assistant asks multiple calls. The user can request that their assistant place 
"Where?" To this the subscriber may respond with one of the different callers on hold, hang up on certain callers, and 
following designations: work, home, car, mobile or other, place calls to other contacts through the feature phone, 
for each of which it is assumed that a correspodning address When a subscriber receives a call using the system's smart 
or phone number exists. 65 call-waiting, the call is managed by the feature phone. 

After the subscriber has indicated how he will be Current implemented functionality includes the user com- 

reachable, the electronic assistant prompts him to indicate mands: call, hang-up, and press-the-hold button. These 
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commands are used to manipulate an unlimited number of contacts) use the reception assistant. As such, once the 

simultaneous calls. extension for the subscriber is entered the subscribers assis- 

Virtual Hallway tant immediately asks for the subscriber's passcode. Since 

As indicated above, the subscriber can ask his electronic the assistant can assume it is the subscriber it does not need 

assistant what is in the virtual office, what it is holding in its 5 to ask for the name of the person calling, as previously 

hand or pointing to, what is in the "trash", what new described. 

messages have come in, etc. However, a concept of a virtual T . , , . , . , . ■ ♦ « *i_ 
, „ & . , j , • . A ™ It is possible to also mclude electronic assistants with 
hallway is also supported in the described embodiment. The r „ . „ . . t - 
virtualhaUway is made up of the collection of virtual offices. duTerent personalities i.e., assrstants whose functionality 
The subscriber can ask his electronic assistant what other 10 15 tallor f ? * the P artlculal : ^scnber for whom that assjs- 
subscribers are presently communicating with their elec- tam ^ °e Providing services. For example, there could be 
ironic assistants by using a "Who-Else-Is-Around" com- ™ electronic assistant for customer serv.ce representatives, 
mand. This feature can be "filtered" so that the subscriber That electronic assistant would implement a set of corn- 
will only see people who are in the subscriber's contact list, mands md tasks that are morc appropriate to the role of a 
or who are members of certain specified groups, or who are is customer service representative or even a group of customer 
part of a particular phone book. representatives. It might handle an incoming call by 

In the described embodiment, which supports internal V*** ""f a « of.^estions designed to gather 

switching, and given that the virtual hallway feature enables "^formation relaUng to the particular customer service prob- 

the subscriber to see other subscribers who are on the lem - 71,21 ""formation might be used by the electronic 

system, the system also offers the capability to connect and 20 « MSta ^ 1 1 ° *termine the most appropriate person to receive 

share information between distributed or travelling groups me caU and ".^S^ ako be used to assist the customer 

or subscribers more quickly than is possible by messaging. representative in responding to the customers concerns by 

In addition, "visibility" in the hallway can also include the retrieving relevant stored data from a database. The elec- 

ability to see what the other person is doing (e.g. on the » Btant P lace me caUer . m a ^ ^ other 

phone, reading a message, etc.). Depending upon what the 25 c ^ crs havm g service questions and then connect 

subscriber has told her assistant about her availability and 4(5111 resources become available, 

accessibility, the electronic assistant will control how visible In addition to "customizing" the personality and skills of 

the subscriber is (or how visible she will be) in the virtual a subscriber's electronic assistant other specialized elec- 

hallway. tronic assistants can be added to the system. For instance, a 

Implemented Assistants 30 system may have, or be able to connect with, a stockbroker 

The described embodiment implements two types of assistant to check on particular investments, 

assistants, namely, the electronic assistant described above Other embodiments are within the following claims. 
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- Messages 6371 
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- Take a message 8 
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- Phone book listing 725 
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Hang up 487 


- Reminders 736 


What's it say? 934 




- Tutorials 888 


Wildfire 945 




Good-bye Wildfire 429 







and a reception assistant. The reception assistant is much 
simpler than the electronic assistant, its job is to answer a 
central phone number (for all of the subscribers on a 
particular system) and route the call to the assistant for a 
specific subscriber. This avoids using the DID line and 60 
allows one number to handle a large number of subscribers. 
The basic operation of this assistant is to ask for the 
extension of the subscriber being dialed and then start up 
that subscriber's assistant to handle the rest of the call. At 
this point the call can be handled in a similar manner to the 65 
answer call task previously described. In the described 
embodiment, it is assumed that only subscribers (not their 



What is claimed is: 

1. A method implemented by a computer-based electronic 
assistant for managing information and connection 
resources for at least one subscriber, said method comprising 
the steps of: 

establishing a first connection between the electronic 

assistant and the subscriber; 
identifying the subscriber; 

starting a first session between said electronic assistant 
and said subscriber, said first session being a thread of 
execution of code for managing data and performing 
functions on behalf of said subscriber, 
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within said first session, receiving a first command sent by 
the subscriber to the electronic assistant over the first 
connection, said first command instructing the elec- 
tronic assistant to perform a function, 

within said first session, responding to said first command 
by performing said function, said function including 
managing said information and connection resources as 
directed by said subscriber, and 

switching said electronic assistant between a foreground 
mode of operation in which said subscriber interac- 
tively instructs said electronic assistant to perform 
functions, and a background mode in which said elec- 
tronic assistant monitors the subscriber connection and 
responds to a limited, smaller set of commands. 

2. The method of claim 1 further comprising the steps of 
receiving a call from the subscriber to the electronic 

assistant over a communications media; 
in response to receiving the call from the subscriber 
establishing said first connection. 

3. The method of claim 1 further comprising the steps of 
receiving an incoming call to the subscriber from a caller; 

in response to receiving the incoming call, establishing a 
second connection between the electronic assistant and 
the caller; 

over the first connection, electronically notifying the 

subscriber of the incoming call; 
in response to receiving a call accept command from the 

subscriber over the first connection, linking the caller 

and the subscriber so that they can communicate with 

each other; 

upon linking the subscriber to the caller, switching the 
electronic assistant to the background mode in which 
said electronic assistant continues to monitor the sub- 
scriber over the second connection while the subscriber 
is linked with the caller; and 

in response to receiving a summoning command, switch- 
ing the electronic assistant into the foreground mode. 

4. The method of claim 1 further wherein said establishing 
step operates in response to the steps of 

recognizing, by said electronic assistant, a requirement to 

establish said first connection, and 
further comprising the steps of 

issuing an electronic request by the electronic assistant to 
the subscriber, and 

recognizing and acting upon the response by said sub- 
scriber to said electronic request. 

5. The method of claim 1 further wherein said establishing 
step comprises the steps of 

executing a locator algorithm to determine the likely 
whereabouts of the subscriber at a specified time; and 

if the locator algorithm determines a location at which the 
subscriber is likely to be, identifying a communications 
device that is associated with that location, 

wherein the communications device that is associated 
with that location is the communications device to 
which said first connection is established. 

6. The method of claim 1 further comprising the steps of 
storing an electronic reminder in an electronic database 

that is accessible to the electronic assistant, said elec- 
tronic reminder being addressed to said subscriber, and 
said electronic reminder including subscriber- 
generated content and a specified time at which it is to 
be delivered to the subscriber; 
when current time coincides with the specified time, 
detecting that the stored electronic reminder has 
become due; 
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in response to detecting that the stored electronic 
reminder has become due, identifying a communica- 
tions device through which the subscriber can be 
reached at the specified time; and 

said establishing step establishing a connection to the 
communications device. 

7. The method of claim 6 further comprising the steps of 
upon reaching an answering party through the communi- 
cations device, electronically notifying the answering 
party that the call is intended for the subscriber; 

electronically informing the answering party that the 
answering party may accept the call by issuing an 
accept reply; and 

if the call is accepted by the answering party, electroni- 
cally delivering the contents of the electronic reminder 
to the answering party through said communications 
device. 

8. The method of claim 6 further comprising the steps of 
delivering the contents of electronic reminder to the 

answering party, 

maintaining the connection to said communications 
device so that the subscriber can issue commands to the 
electronic assistant; 

monitoring the connection for subsequently subscriber- 
issued commands; and 

in response to receiving the subsequent subscriber-issued 
commands, performing other operations for the sub- 
scriber. 

9. The method of claim 1 further comprising the steps of 
receiving an incoming call to the subscriber from a caller; 
in response to receiving the incoming call, establishing a 

second connection between the electronic assistant and 
the caller; 

through a dialogue between the electronic assistant and 
the caller over the second connection, determining the 
identity of the caller; 

electronically alerting the subscriber over the first con- 
nection that there is an incoming call for the subscriber; 

electronically identifying to the subscriber the identity of 
the caller; and 

monitoring the first connection for a response sent by the 
subscriber to the electronic assistant directing the elec- 
tronic assistant how to process the incoming call. 

10. The method of claim 1 wherein said step of perform- 
ing said function comprises the step of performing a 
CREATE-A-CONTACT function. 

11. The method of claim 1 wherein said step of perform- 
ing said function comprises the step of performing a 
REMIND-ME-TO-CALL function. 

12. The method of claim 1 wherein said step of perform- 
ing said function comprises the step of performing a 
REMIND-ME function. 

13. The method of claim 1 wherein said step of perform- 
ing said function comprises the step of performing a TAKE- 
A-MESSAGE function. 

14. The method of claim 1 wherein said step of perform- 
ing said function comprises the step of performing an 
TLL-TAKE-IT function. 

15. The method of claim 1 wherein said step of perform- 
ing said function comprises the step of performing a HOLD- 
ALL-CALLS function. 

16. The method of claim 1 wherein said step of perform- 
ing said function comprises the step of performing a CALL 
function. 

17. The method of claim 1 wherein said step of perform- 
ing said function comprises the step of performing a FIND 
function. 
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18. The method of claim 1 wherein said step of perform- 
ing said function comprises the step of performing an 
I-WILL-BE function. 

19. The method of claim 1 wherein said step of perform- 
ing said function comprises the step of performing a NEV- 
ERMIND function. 

20. The method of claim 1 wherein said step of perform- 
ing said function comprises the step of performing a SEND- 
A-MESSAGE function. 

21. The method of claim 1 wherein said step of perform- 
ing said function comprises the step of performing a SEND- 
A-PAGE function. 

22. The method of claim 1 wherein said step of perform- 
ing said function comprises the step of performing a SEND- 
A-REPLY function. 

23. The method of claim 1 wherein said step of perform- 
ing said function comprises the step of performing a TELL- 
ME function, 

24. The method of claim 1 wherein said step of perform- 
ing said function comprises the step of performing a THAT- 
WI LL-BE- ALL-FOR-NOW function. 

25. The method of claim 1 wherein said step of perform- 
ing said function comprises the step of performing a WHAT- 
ARE-MY-OPTIONS function. 

26. The method of claim 1 wherein said step of perform- 
ing said function comprises the step of performing a 
WHAT S-IT-S AY function. 

27. The method of claim 1 wherein said step of perform- 
ing said function comprises the step of performing a 
WHERE- WERE- WE function. 

28. The method of claim 1 wherein said step of perform- 
ing said function comprises the step of performing a WHAT- 
ARE-YOU-HOLDING function. 

29. The method of claim 1 wherein said step of perform- 
ing said function comprises the step of performing a DATE- 
IT function. 

30. The method of claim 1 wherein said step of perform- 
ing said function comprises the step of performing a 
GO-BACK function. 

31. The method of claim 1 wherein said step of perform- 
ing said function comprises the step of performing a 
THROW-IT-AWAY function. 

32. The method of claim 1 wherein said step of perform- 
ing said function comprises the step of performing a SEND- 
A-COPY function. 

33. The method of claim 1 wherein said step of perform- 
ing said function comprises the step of performing a 
PREVIOUS-ITEM function. 

34. The method of claim 1 wherein said step of perform- 
ing said function comprises the step of performing a NEXT- 
ITEM function. 

35. The method of claim 1 wherein said step of perform- 
ing said function comprises the step of performing a GIVE- 
THEM-A-CALL function. 

36. The method of claim 1 wherein said step of perform- 
ing said function comprises the step of performing a FIRST- 
ITEM function. 

37. The method of claim 1 wherein said step of perform- 
ing said function comprises the step of performing a FILE- 
IT function. 

38. The method of claim 1 wherein said step of perform- 
ing said function comprises the step of performing a 
DESCRIBE-IT function. 
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39. The method of claim 1 wherein said step of perform- 
ing said function comprises the step of performing a FIND 
CONTACT function. 
5 40. The method of claim 1 wherein said step of perform- 
ing said function comprises the step of performing a FIND 
ALL-THE-CONTACTS function. 

41. The method of claim 1 wherein said step of perform- 
ing said function comprises the step of performing a FIND 

10 NEW-MESSAGES function. 

42. The method of claim 1 wherein said step of perform- 
ing said function comprises the step of performing a FIND 
NEW-MESSAGES-FROM function. 

1S 43. The method of claim 1 wherein said step of perform- 
ing said function comprises the step of performing a FIND 
MESSAGES function. 

44. The method of claim 1 wherein said step of perform- 
20 ing said function comprises the step of performing a FIND 

MESSAGES-FROM function. 

45. The method of claim 1 wherein said step of perform- 
ing said function comprises the step of performing a FIND 

25 FILED-MESSAGES function. 

46. The method of claim 1 wherein said step of perform- 
ing said function comprises the step of performing a FIND 
PHONE-BOOK-USTING function. 

30 47. The method of claim 1 wherein said step of perform- 
ing said function comprises the step of performing a FIND 
REMINDERS function. 

48. The method of claim 1 wherein said step of perform- 
35 ing said function comprises the step of performing a FIND 

TUTORIALS function. 

49. The method of claim 1 wherein said step of perform- 
ing said function comprises the step of performing a GOOD- 
BY function. 

40 

50. The method of claim 1 wherein said step of perform- 
ing said function comprises the step of performing an IT'S 
ME function. 

51. The method of claim 1 wherein said step of perform- 
45 ing said function comprises the step of performing a 

DESCRIBE-IT function. 

52. The method of claim 1 wherein said step of perform- 
ing said function comprises the step of performing a 

50 UPDATE-IT function. 

53. The method of claim 1 wherein said step of perform- 
ing said function comprises the step of performing a 
TRANSFER function. 

55 54. The method of claim 1 wherein said step of perform- 
ing said function comprises the step of performing a 
CREATE-A CONFERENCE function. 
55, The method of claim 1 further comprising the steps of 
60 receiving an incoming call to the subscriber from a caller; 
in response to receiving the incoming call, 
establishing a second connection between the electronic 
assistant and the caller; 

65 through a dialogue between the electronic assistant and 
the caller over the second connection, determining the 
identify of the caller; 
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electronically alerting the subscriber over the first con- 
nection that there is an incoming call for the subscriber; 

electronically recording for the subscriber the identity of 
the caller; and 

monitoring the first connection for a response sent by the 
subscriber to the electronic assistant directing the elec- 
tronic assistant how to process the incoming call. 

56. The method of claim 55 further comprising the step of 

recognizing the electronically recorded identity of the 
caller. 
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57. The method of claim 56 further comprising the steps 

f 

determining whether the recognized caller meets stored 
criteria, and 

announcing the recognized caller when said criteria are 
satisfied, and 

screening the recognized caller when said criteria are not 
satisfied. 
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